Dreaming In Code

by Scott Rosenberg

"Software is hard," Donald Knuth wrote many years ago, when software seemed much simpler. This brave book tries to explain how hard software is, to an audience that knows nothing about software and believes their ignorance to be a virtue and a right. People don't know exactly how their cars work, but everyone has opened the hood and seen the engine whir and watched wheels move. Lots of people have no real idea what programming is, or why it's hard.

Rosenberg clearly planned this to be the software Soul of a New Machine, Tracy Kidder's memorable exposé of life on the cutting edge of computer design. He chose for his target a sure thing: Chandler, a personal information manager instigated and funded by Lotus founder Mitch Kapor. With ready access to ample funds, management expertise, the power of open source, and some of the best software minds in the industry, Chandler must have seemed the place to be.

Chandler sure scared me, since it was announced shortly after Eastgate committed to Tinderbox, stems from one of the same inspirations — Kapor's agenda— and sought to address some of the same problems. Because Chandler overlaps so much with Tinderbox, I find it difficult to write about the project with confidence.

Vexingly, Chandler didn't work out: by the time Rosenberg decided to finish the book, Chandler had bogged down in protracted code thrashing. But we don't know that there are no second acts in software; perhaps Chandler is going to emerge next week and be terrific. Or perhaps someone will fork it and save the project. Or perhaps Tinderbox made better technical tradeoffs and created a useful modern agenda and that's really what was needed.

Rosenberg makes clear that Chandler suffered from a baked-in incoherence: it was a successor to Kapor's agenda and so it was about agents and lexical properties of notes, but it was also supposed to leverage ubiquitous email and calendaring, and so wound up being about dates and places. That's too much to fit into a product, and there's a Microsoft blockage astride the calendar/email track. Not recognizing this was probably an irremediable mistake. The hybrid Python/C++ architecture always looked dicey to me, as did the premature commitment to be cross platform at launch. But hindsight is easy (and that hybrid architecture looks better now than it did in 2001).

On this framework, Rosenberg hangs a masterful and engaging survey of the thinking that underlies contemporary software engineering. This overview will have lasting importance, as I think it's destined be the textbook that introduces a generation of students throughout the world to the professional practice of software and to its founding voices — Brooks and the Mythical Man Month, Parnas, Joy, the postmoderns, the agilists.

What Rosenberg doesn't capture — because Chandler seldom captured it — is the way software actually gets written: in slow, steady segments, in dashing sprints, in long nights of inspiration, in weeks of staring at the screen, but always — in the end — by one or two people working to get something to work. In practice, this usually means one or two people imagining how it might work, and then making it happen. There wasn't enough of this in Chandler, and when it did happen, it too often happened to infrastructure, deep in foundations that were expected to underpin grand structures that were never built.

Business writers tend to attribute corporate success to the genius of a CEO, and to blame losses on that executive's personal foolishness and depravity. This is sentimentality, not journalism, and its prevalence is a symptom of the corruption of our business press.

Rosenberg set out to emulate a terrific book about hardware in order to explain software. He chose wisely. He had terrific access. He was spectacularly unlucky; instead of covering a technical revolution, he had the inside story of a project that fizzled. He brilliantly avoids focusing on the failure, the consequences, the recriminations; instead, he explores the underlying needs and explains why failures like this happen all the time. To have carved such a fine, generous, and useful book out of the debris is a very fine accomplishment indeed.