A very long time ago, I worked at Motorola. And I was incredibly proud to work there, and met many brilliant people. And some, less so.
At the time, the working assumption in the business world was that Japan’s culture of cooperation, hire for life, logistics and quality techniques would take over every industry, with supercomputers being next. Soon. America was dead because our cars were crappy.
Our cars were crappy.
Motorola responded with a program called Six Sigma, which revolved around aiming for 3.4 defects per million units. Ironically, Motorola already had a well-deserved reputation for excellent quality in both hardware design and manufacturing. We didn’t have a quality problem, we had a cost of goods problem. One that Nokia would soon exploit and do fatal damage to Motorola.
Of course, Six Sigma had to be applied to software too, since a good program is good program and should be used everywhere. And no one had a clue how to do it, but we saluted the flag and wrote things about it that were purely fictional and didn’t ask too many questions.
A few years into that job, I noticed that we were building a lot of clever, complex, high quality, expensive products that were failing in the marketplace. I had no idea why, and clearly our leaders didn’t either. Down the road, Microsoft was shipping obvious cheap junk and people in their late 20s were buying 911s at lunch. I few years later I would work there and learn why their products were doing so well.
In 1988, Motorola won the first Malcolm Baldridge award given by the US federal government for American quality.
I read seemingly endless industry press, and internal preening, about how Motorola’s practices made us an almost invincible force in technology.
I learned a few important things from that experience:
Companies (later, I learned all organizations), even formerly great ones, at scale and given time, lose their edge. They do what worked before, and hope it works again. When cellular phones with a bill of materials one half of ours appeared, the company was on the path to extinction. And a manufacturing process designed by committee and implemented by The Dept of Bureaucrats might not be the best way to compete in a rapidly moving new software industry.
Companies can be far down the road to extinction, while making huge amounts of money selling old stuff, with the damage being only barely visible inside the company, and invisible outside the company. The next story the press writes is “nobody could have seen it coming”, followed by a bunch of incorrect analysis about what happened.
A long time ago, having worked at Microsoft for only a few years, I helped ship an extension to Windows 95 that implemented VPN. And after shipping it, we went back to Nova Scotia for vacation. While there, I discovered that the shipped feature wouldn’t maintain a connection back to Microsoft in Redmond for longer than a few minutes, when it just stalled forever. We hadn’t seen this in testing.
I debugged it there, and tracked it down to two things: a bad exponential backoff timer – it only got larger, never smaller – in TCP/IP that had been in 9x since ship, and poorly designed VPN protocol that added huge amounts of latency to every packet, ironically, because of the TCP/IP issue in the outer tunnel layer impacting the inner tunnel.
That’s pretty funny if you are a nerdy dweeb.
Anyway, I was horrified. We had already had a million or so downloads of this thing, and that was a scary big number back then. I had shipped something that worked perfectly on our best in class internal networks, but didn’t work in the real world of modems and bad ISPs.
Would I get demoted? Yelled at? Mocked? Would it end my career? Read about it in press? Would my mom find out?
None of this of course happened. In fact, it was a huge effort to get a fix shipped long after it was done. Nobody cared, because there was no feedback loop. There was no channel to collect enough samples of issues to collate and sort them, outside of our internal testing which didn’t work in this case. And we were steamrolling everyone in the industry and making money hand over fist.
Obviously, validation that we were the smartest people, with the best strategy, execution and products and that was that.
What was happening, of course, is that the core value proposition of Windows, that it ran every program on every piece of hardware, combined with the innovation and economics of a high scale hardware industry overwhelmed everything else. The model, and the product, changed the world forever and create one of the most important companies in the last 100 years.
In those days, in the absence of a working, scale, feedback loop, we “dogfooded” our own products ruthlessly. Basically, we ate out own dogfood to see if it was tasty and nutritious. We were our own testers, and we tested the design, the models, and the implementation. And checking in shit was frowned on. Oddly, once the shit was checked in, fixing it properly was frowned on. And using that method, we shipped Windows 95, Windows NT, Visual Basic, Access, Office and many other products that changed the world and continue to do so today.
But eventually scale and 50,000 new hires would overwhelm that method, and something better was needed. We listened to the top experts in the world, and learned that, despite localizing and selling our products in every language and every country in the world, including right to left languages, and built-in support for the hearing and vision impaired, we were too stupid, bigoted and physically large to understand that our game controllers were too big for Japanese hands. The solution, somewhat counter-intuitively at least to me at the time, was to change our hiring, promotion and firing process to be structurally more favorable to women, and to a lesser degree, Blacks, so that “we looked like our customers”.
Something that is now so core to the company’s mission that the CEO is paid for performance on it:
What a relief that the solution was so simple.
Of course, we also had to build and empower The Dept of Bureaucracy to make sure everything was right, and worked right, and we never made any mistakes.
The company was hated for its arrogance, greed, and bad products.
This of course has all been turned around now. Microsoft is making money hand over fist selling old stuff, and the press and shareholders love them. They love themselves, like we used to, in the good old days. People in their late 20s are buying Teslas.