Chapter 1. Introduction

A FEW YEARS AGO, I ATTENDED AN IBM RATIONAL UNIFIED PROCESS TRADE SHOW, HELD AT A CONVENTION center somewhere just outside Dallas. The center was a sprawling complexhotels, restaurants, convention hallsanchored by an enormous indoor arboretum, a five-story glass-domed cathedral landscaped with palmetto and live oak, featuring, on its west wall, an exact replica of the Alamo, only at about twice the size.

Naturally it was a Disneyland-clean version of the Alamo. And the stone pathways, trickling canals, and set-aside grottos were all exquisitely manicured to reflect a kind of comfortable glow balanced between true life and cartoon. The theme of the show was something like, "Achieving Your Quality Goals." As I looked around, I knew this place was the perfect choice for that. Whoever had taken on this landscape project had not let up until every blade of grass had been trimmed to shape, oiled green, and shellacked properly into place.

This was Wednesday evening, after the show's opening, and the sponsors were hosting a huge cocktail reception in this arboretum. So after an afternoon of moving in and out of sessions and giving a brief talk on tailoring CMMI to organizational size, I found myself in a packed crowd in front of the Alamo, standing just under St. Jerome, at the best bar in the place. That's where Scott Brenner introduced himself.^[*]

^[*] The name has been changed. Throughout this book, I'll be citing stories and case histories that illustrate many facets of process improvement. Most of the names and companies I cite are real. You'll probably recognize them. But when an event might be embarrassing or the individuals involved have specifically requested I not use real names, I use pseudonyms. The details, however, are all as accurate as I know them to be.

Scott was a senior IT director for a major health-claims processing company, one of the largest in the nation, and he wanted my thoughts on how he might improve the way his people managed IT projects. It seems that he and the company had had a "pretty rough" year last year getting his folks marching along productive lines. Scott didn't know a lot about process improvement, but his company used a lot of the Rational products and, given his role in the organization, management thought he'd be a good choice to send out into the field to collect some ideas. We chatted for a few minutes about the size of the organization, how it was structured, and what kinds of work its people engaged in. From Scott's answers, it seemed like a pretty typical Fortune 100-type shop. Then I asked him if he had any ideas about what specifically he might want to improve.

"I guess," he said, "it'd be our drop rate."

I'd never heard of that. "What's a drop rate?"

"That's the return revenue we lose when our systems misapply or misdirect a payment claim."

I liked that. That was a tangible metric. And I liked the fact that Scott seemed to have a handle on it. This seemed like a pretty good start toward a meaningful improvement target.

"Do you know what your drop rate was last year," I asked.

Scott took a club soda from the bartender. "Maybe three hundred million."

I knew I heard him right because the shooting didn't start until just after that. In fact, I'm pretty sure the "three hundred million" may have been what started it. But whatever it was, Santa Anna's men were suddenly swarming all around us, musket balls whizzing through the air, the glint of fixed bayonets throwing off slivers of hard light. I could sense The Defenders somewhere above us, stalwart shadows moving against the wash of cannon smoke, firing bravely over the crest of the chapel. And here we were, right out front. How did we manage this? Had they forgotten us? Could we ever hope to make the wall? Or were we lost? Were we doomed? Was it all a futile...

"James?"

I blinked. Scott was back in front of me, making a slight gesture with his club soda. The ice clinked against the glass.

"Right," I said. "Absolutely. Project management. Yeah. Well, there are certainly things. Several things, actually. And things for drop rates...." And I can't really remember what else I said. But I'm reasonably sure it made little contribution to the industry's overall body of knowledge. I was stuck on that $300 million figure. To me, that's a fantastic amount of money to misapply or misdirect. And to make things more fantastic, I knew this company. I owned its stock. A good bit of it actually. And I hadn't heard of any such debacle. Not in the papers. Not from the analysts. In fact, when I checked later, I found that for that "pretty rough" year he was talking about, the stock had actually gone up two percent.

I knew this kind of thing happened, but I had never been this close to it happening in such a big way. Another technological potholethis one about the size of the Astrodomecloistered behind the walls of American IT.

As a rule, corporations like to keep their troubles quiet. They prefer to keep their problems from showing up on the street. It can be embarrassing. And it takes a lot of explanation to a lot of people who probably aren't prepared to appreciate that these things happen. Nobody's perfect. Defects happen.

Meat packers and car manufacturers probably have it the worst. The defects that enter their products are often miniscule in size: a bit of bacteria here, a missing shim there. And so they can be hard to spot. Often it's not until the consumers themselves stumble on it (a case of food poisoning, a crushed bumper) that the problem emerges. When that kind of failure goes public, an organization will scramble to locate the source of the problem. The company will divert all kinds of energy to tracking down the problem's cause and determining how much of the defective product got out. The course of last resortwhen the milk's really been spiltis usually to set a recall into action. Take your ground beef back to the supermarket; return to your dealership. Recalls are usually massive undertakings, and they can cost millions and millions of dollars. But by the time the error's in the public eye, the company has no real choice. Public failure is always expensive.

But sometimes it's also purging.

Take the Chevrolet Venture for example. This popular minivan ran into something of a quality problem in the late 1990s. At issue was the questionable design of its passenger seat latch. The company received data that indicated maybe the latch had an operational defect. After some study, Chevrolet determined that a general recall of the 1997 Venture was in order. Here is an excerpt from the text of that recall notice: "RECALL NOTICE (1997 Chevrolet Venture): Passenger's fingers could be severed by latch mechanism that moves seat fore and aft (Consumer Reports 2005)."

Now, as a general rule, severed fingers should not be a part of anyone's driving experience. What manufacturer would want that picture etched into the brand image of their minivans? Chevrolet no doubt brought its forces to bear. That latch was probably redesigned and remanufactured against the highest standards, with inordinate attention to detail: test after test, exhausting verification runs, complete assurance of unquestionable integrity. It wouldn't surprise me if that latch todayalmost 10 years laterstands as the world-class epitome of efficiency and safety in seat latch technology.

That same call to quality came to poultry processor Pilgrim's Pride in April of 2002. In the largest meat recall in U.S. history, the company pulled back 27.4 million pounds of cooked sandwich meat after internal tests came back positive for potential listeria contamination. The discovery followed outbreaks of listeria in eight Northeast states that caused at least 120 illnesses.

Pilgrim's Pride, with encouragement from the FDA, left no tile unturned in its quest to remove even an amoeba's chance of impurities finding home in its food sources. There was simply too much riding on the outcome. Today, the company has one of the most stringent processes for meat inspection, meat processing, and meat packing in its industry.

Chevrolet and Pilgrim's Pride are both companies with established reputations. They have not grown to who they are by ignoring issues of product integrity, product performance, and quality. However, the complex nature of their businesses means things will not always go as planned. But with each misstep comes improvement: things get better, and then a little better. In these cases, the companies followed, but the public led. The public, when it's all said and done, may be the most effective quality-control system anyone could have.

That brings us to one focus of this book: performance improvement in technology companies.

No one screamed at Scott Brenner's company, because no one outside knew about the fault points of its claims systems.

From my experience, technology industriescorporate software, systems development, and operationshave been somewhat immune from the cleansing light of public failure. It's common in this industry that usersused to less-than-perfect technology productswill often put up with buggy software. They learn to work around the problems or to find substitute solutions while the issues are being fixed.

Most of the major missteps that occur on technology projects are discovered long before the end product ever sees real-world deployment. People internal to the organization observe that the new payroll system is printing security-sensitive data on checks. State taxes are being calculated at erroneous rates. Systems slow down to a crawl when new functionality is plugged in. The lights go out. Maybe we should back up a step.

In a way, that's fortunate: some of these potentially disastrous production problems are avoided. But by another way of thinking, it might also be unfortunate. When a project goes up in smoke, no one outside a core group may learn the details. These failuresoften by publicly held companiesare all too often discreetly swept under the rug, or quietly navigated back to the starting gate with a no-harm-no-foul attitude. Often millions of dollars have been spent by the time the plug is pulled. Millions of dollars down the drain, with little to show for it.

And then there are the potentially disastrous production problems that do get through. Some folks might say, "Sure, but we aren't talking road safety here, or botulism." But given the prevalence of technology today, we just might be. Anyway, the issues certainly belong in the same ballpark.

The general feeling, as I have gathered it firsthand in the technology field, is that these miscalculations, false starts, and first shots are simply part of the cost of doing business. These situations (I have seen my share of them) are often further rationalized with a slew of caveats: we were never given the right resources, the timeline was always unrealistic, the specs were never clear. And because many corporate managers outside of IT assume that what goes on inside IT must be techno-mystery, the level of questioning that might penetrate these veils is seldom undertaken.

As a consultant, I have seen this happen in big ways and in small ways at MCI/WorldCom, Home Depot, GTE, Public Health Software Systems, ALLTEL, and many others. Ten years ago, the Standish Group released what is now famously known as "The Chaos Report." This was a study of software projects undertaken by Fortune 500 companiesthe big players in the game. The results showed that the average project managed by these guys ran 222 percent over schedule and 189 percent over budget, with over 30 percent of planned functionality missing in the final product. Those were 1994 figures. About three years ago, Standish updated its Chaos Study, and though the numbers had improved somewhat, the deficits were still considerable.

I doubt there was a public recall in the bunch.