Industrial Anti-patterns

being those ways not to construct a software industry, or, some consequences of time spent in handbaskets.

(Incidentally, I have fooled with a comments widget, but it doesn't seem to actually go the to appropriate comment quite yet. I'm not sure why not.)

Some time ago I ranted at some length and some heat about the state of software development - or, to be exact, about a particular approach to teaching software development and the immediate consequences of the worst of it. Allow me to generalize somewhat:

All software engineering techniques proposed to date have failed. All seriously considered approaches to sidestepping this issue have failed.

As evidence of which, allow me to start with Windows Vista. Whatever you may personally think of Microsoft's software or business ethics, it is unquestionable that Microsoft has had as much impact on computing today as any company ever, and that Microsoft has made amounts of money comparably obscene with any other computing company ever. It is probably also safe to assume that Microsoft employs a large number of smart people. Given all that, it is fair to ask what it means when such a well funded, well equipped company cannot produce a new version of their major money-maker without falling six years behind, overspending their budgets by I don't even want to imagine how much, rewriting the whole thing at least once, shuffling their executive leadership twice (counting the post-Windows 2000 shuffle that got the whole Vista game off the ground), and finally producing a product that is, from my beta testing experience, about two years of improvement over 2001's Windows XP.

If you would prefer examples besides Microsoft, consider the gaming industry, which cannot release a major software or hardware project without release-day patches, bugs, replacements, etc. Or consider Acrobat Reader, which is quite possibly the most bloated, over-complicated piece of software on my desktop, and has probably convinced me not to use any other pieces of Adobe software ever. Or consider Firefox 2.0, which not only added zero compelling (to me) features over Firefox 1.5, but made all my extensions not run any more. At least when I first saw IE 7 I said "hey, there's some cool stuff here." When I saw Firefox 2.0, I said "Where's my Firefox 1.5 installer?"

Or consider much corporate IT. Then cry.

At some point, I claim, it is worth asking if we are seeing more than individual anomalies; if perhaps the success stories are the exceptions instead of the rules. If so, we ought to be able to find some root causes of our current state.

My claim is that there are not enough good programmers to complete the number of large software projects being attempted with the tools available to them. This has two obvious remedies:

First, those programmers can attempt to do so anyway. This results in overwork, burnout, and said programmers doing significantly less than their best work because they don't have the mental energy left. From what I've read, this is much of the problem in the gaming industry.

Second, you can hire a number of poor programmers to supplement the good programmers. This serves as a demonstration of the truth of various trite sayings about weakest links. Oh, and The Daily WTF again.

Finally, allow me to briefly propose an almost Marxian justification of the development of information technology. Rather than seeing a smooth evolution of computing technology, the industry seems to be defined by generally smooth, but slow development, punctuated by disruptive jumps in technology which change the structure and techniques of IT significantly. The spread of PCs and replacement of mainframes, the spread of client-server methodologies, and the return of thin-client approaches can all be seen as these kinds of changes. In each case, the computing profession was changed significantly in the process. For example, the spread of PCs removed the need for the highly structured and segmented groups of people responsible for the maintenance of mainframes. The spread of thin-client techniques similarly made it possible for programming to distance themselves from the desktop environment, and enabled the spread of Java, Ruby, etc.

I believe we are at the brink of another such change: the coming of multicore processors. Heretofore, threaded programming has been awkward and generally the province of specialized engineers at places like Microsoft and Oracle. The majority of corporate IT could get away with writing single threaded code and occasionally relying on a convenient piece of server software to handle multiprocessor machines. As multicore development continues (and Intel is threatening 80-core machines in the near future), this approach will be increasingly untenable.

In the wake of this transformation, the IT industry will be forced to adapt. It will be unacceptable the business decision makers to pay for upgrades when their capability cannot be used by their IT departments; I suspect the claim that the IT department is simply unable to use the new technology will be equally unpalatable. Yet, the industry seems relatively clueless about how to approach the problem. There are rumblings in places like Microsoft Research - leading to Software Transactional Memory in its various forms - but none of these are close to being distributed in Visual Studio. The gaming industry, generally at the forefront of using new technology, has been generally frustrated by the multicore Cell machines Sony is selling, and Valve's excitement about having the first multi-threaded 3-d engine (although it hasn't been released) should be some indicator about the usability of multicore machines in PC gaming.

The IT industry has come to a point where its engineering techniques have no way to cope with the coming technological change. This catastrophic failure will not by fixed by rearranging deck chairs - or even running around with a rivet gun and a bucket. We are advised to learn to swim.