Hardware Control Experience with ExoTech Principles
About twenty-five years ago, I was involved in a project that designed and implemented what we would today call a specialized hardware-control ExoBrain. Of course, at that time I didn’t know it as an ExoBrain. I only came to that realization after studying what an ExoBrain is and the ExoTech principles it is based on.
The project was enormously successful in many ways and ultimately resulted in a significant competitive advantage for my employer, a large multibillion-dollar company that made disk drives. The competitive advantage stemmed directly from the powerful principles of ExoPonents and Any-to-Any machines. This is a real-world example of how they manifested. But before I get to that, I need to provide a little bit of technical background.
Embedded Firmware
Most people only know that disk drives store their programs, documents, spreadsheets and other data that their computer uses. To explain my assertion about this specialized ExoBrain, I need to explain a little bit about what is embedded in disk drives, something called firmware.
Firmware is a type of computer software or set of instructions stored on a persistent memory chip. When connected and applied to an integrated computer system designed to control a specialized device it is often referred to as embedded firmware that is part of an embedded system. “Embedded” because it is fully contained within a complete device composed of additional pieces of hardware (electronic, electrical, mechanical).
A disk drive contains an embedded system controlled by firmware. This firmware is enormously complex and multifaceted, but any one instance of it has only one purpose. When a version is loaded for a customer, it basically stores and retrieves data reliably, communicating through a hardware interface to a connected computer or hardware network according to that customer’s standards. Prior to any such “customer code,” a version is loaded for the manufacturing process where most of what it does is execute a self-contained program that tests, adjusts and configures all the subsystems to operate properly per specifications and then logs the results for later analysis. That version of firmware, like all firmware these days, is a large, monolithic program and suffers from all the same maladies understood in ExoTech that plague state-of-the-art software in general.
A New Paradigm
Back in the early 1990s, those first versions of manufacturing firmware were simple enough that one firmware engineer could write the code, test it and debug it over the course of a product’s development, maybe nine months or so of work. However, in the years that followed, more demand was being placed on the firmware, while at the same time each new product typically had a different processor hardware platform, meaning the code had to be rewritten almost from scratch each time. Add to that the continuous increase in complexity of disk drives, and soon small teams of engineers were needed to create and maintain the manufacturing firmware.
My job at the time was to design, test and qualify the manufacturing process for each new product developed at the site I worked at. I had to produce the specification that the firmware engineers needed to write their code. In essence, I was their customer and I was in a position to see the progression of increased difficulty and problems they were having. There came a time when I wanted to absorb into the manufacturing firmware nearly all external parts of the test process while at the same time expanding the number of required processes. That amount of increased functionality broke the model that was being used to create firmware up to that point. A new model had to be developed.
By whatever confluence of skill and luck, our team, consisting of about a half dozen engineers, managed to settle on a design that did not rely on a monolithic approach to programming the many processes and steps required. Instead, we broke down each of the internal drive systems into functional components and kept breaking them down into subparts until they could not be broken down further without losing useful meaning within the context of the environment that they needed to operate in. The list of parts was what ExoTech would identify as (code) ExoPonents.
After identifying the components, we then figured out how to connect them in useful ways on the fly by creating our own internal interpreter language that integrated access to all the components. We were blessed with an engineer who had done a similar type of thing earlier in his career and he was able to implement the interpreter and components via a very small kernel to operate on the specific embedded system hardware of the drive. (In ExoBrain, there is also a very small kernel and a “BossModule” tells the kernel which data and code ExoPonents are to be used and how they are to be used.)
This gave us the capability to write scripts to execute in the new environment we created. The scripts were compiled, downloaded to the drive and executed. Results were tabulated, formatted and stored via just another set of instructions in the script. Once we tested and validated that all the components were working as expected, we were free to create any process we wanted without changing the firmware.
Initially, I only had a general idea of what the manufacturing process would end up being, but I knew it could easily be figured out as long as the capability was there to do anything in any order with any component. That proved to be the defining power of the system. In the months that followed, I was able to run literally hundreds of variations of manufacturing processes as well as scientific investigations into hardware issues for the other engineering teams.
In the end, we delivered a superior, more efficient manufacturing process and the drive product itself was very successful. But it didn’t end there!
Part of this new breed of manufacturing process was the idea of simple portability. The very next product had a completely different set of microcomputer architecture, but from a process standpoint it was still just another drive. When we analysed the components, we found that it was essentially the same in terms of the environment we needed to operate in. Our chief architect only had to implement a new kernel and thereby connect all the components (ExoPonents) to the new hardware. If my memory serves, he had our system functioning in two weeks on the engineering version of the new drive and I was able to successfully run the same manufacturing script that was at that time running in the factory on the first product. The result was we were months ahead in the development of the new process, able to run on an immature and even problematic hardware platform. In addition to refining the manufacturing process, we spent a great deal of the next few months running experiments for the hardware teams and providing them with huge amounts of data that allowed them to accelerate their own work. That drive design was readied and launched to the factory in record time.
Repurposing the System
This successful pattern continued through many more new products. It nearly came to an end when the company closed our site as part of a company-wide consolidation. In a new location, but with the original team intact, we were asked to “do our magic” yet again on another unique hardware platform.
We were successful porting it, but the hardware never worked as expected and the program was cancelled.
Without a patron to champion this system to other programs, the team broke apart and it seemed for a while it was dead. However, within a couple years a new supporter emerged. A senior manager was tasked with trying to organize and revamp the manufacturing firmware engineering team, which by that time consisted of about twenty programmers. It took that many engineers, apparently, to manage a firmware code base that was inadequate (not meeting the goals), fragile (fix one thing and it breaks another) and not able to develop new features fast enough.
Once briefed by the original architect of the kernel and interpreter, that manager brought me and one other on board to once again make it available on the newest hardware platform. This time though we ran into direct opposition when that big team of 20 programmers was taken over by a new manager and split off into a separate organization. There were then two different paradigms in competition to do the same thing. When we started making serious inroads, complaints started flying and since we were small and they were big and we didn’t have anything ready to demonstrate, they won.
But that is not the end of the story! Our manager rebranded our system as an engineering diagnostic and development tool. We successfully ran this way for another couple years, enabling large numbers of research and development projects that refined and optimized hardware characterizations, in one case cutting three months out of a new product introduction.
Summary of an Era
Alas, all this success met an abrupt end. Even though we were providing great value to many groups and teams within the company as a whole, we were operating under the customer firmware development organization and, from their perspective, we were not contributing any value to them. The system wasn’t part of anything they worked on and they never saw a need to learn much about it. The system was still part of manufacturing firmware and when that group desperately needed more space for their own bloated code, they insisted our system had to be kicked out, and that is how it ended.
It is worth noting that there was a lot of skepticism about the stated abilities of this system by most people when first told about it. Only when we used it to solve one of their problems did they see the value. But even then, they never understood it as anything special; it was just another software tool to them.
Many programmers thought it was a “clever” system. Some criticized it as a toy, not fit for real programmers. Some were often quick to point out that it could never replace “professional” programming. After all, it’s much, much slower than a compiled program when doing the same thing. Well, nobody could argue with that, except to say that it’s the wrong comparison.
That specialized ExoBrain enabled massive amounts of research and refinement of processes in an unprecedented short period of time. There is no possible way that could have been done within the same budgets using standard programmers. And the proof is prior to an ExoBrain coming on the scene the firmware organization failed to fulfill most requests being made for specialized firmware. It was common for other engineering groups to make such requests and just as common for most of those to be rejected or abandoned for being too much of a drain on resources, too complex to deliver when needed or simply not being able to compete with higher priority requests. I would estimate our ExoBrain system increased the number of fulfilled requests about 20X. Unfortunately, that was not a tracked statistic in the organization. It was considered unessential work, a goodwill gesture for the most part. The technology was unceremoniously removed, archived and forgotten.
But the facts are, that while the software created our way ran a bit more slowly (but not so slowly as to materially impact the testing process), our way of doing it enabled software to be created for a new product, which was then tested more comprehensively and readied for market, months sooner than with the old way of creating such software.
When I think back to that time, I recall that a few of us knew we had something unique and powerful. We made valiant attempts to articulate it to ourselves and other groups with limited success. We didn’t have the understandings embodied in the ExoTech principles to help us. But this story shows what happens when ExoTech principles are applied to embedded hardware systems.
Carl Anderson is the Senior ExoArchitect of ExoTech Ltd. Carl studied Physics at Arizona State University, Tempe, Arizona, and Advanced Mathematics and Computer Programming at California State University, Los Angeles, California.
Carl is an accomplished engineer and veteran programmer with more than thirty years of success directing engineers and driving strategies in the data storage industry, generating multi-million-dollar revenues and budget savings. He is a well-grounded and insightful leader dedicated to excellence in collaborative engineering development. He has attained a reputation for incisive teamwork across all stakeholder organizations for planning and deliverables throughout the product development cycle. In addition to being an ExoTech Ltd shareholder, Carl is currently engaged in regular study of lessons based on the ExoTech Patent.