Continual Improvement: The Accelerator Button

The study on capability in a chemical manufacturing plant is surprisingly relevant for software companies. The main lesson is that in sustainable software development, you need to strive for continual improvement while resisting the temptation to focus on features and simply working long hours to meet demands. A software company's business is only as sound as its factory, where the factory is made up of a number of "pumps"; the software that the company produces. You may need to take some of your pumps offline occasionally, and every few years you will realize that your factory is completely different than it was previously because of all the changes made to the pumps. And as with real-world factories, only in extreme circumstances, such as when a disruptive technology is on the verge of widespread adoption or a paradigm shift is about to occur, will you need to completely replace the factory.

Each of the following questions examines some of the parallels between chemical manufacturing plants and software development.

How many developers view writing test code, building testability into their code, enhancing a test framework, or cleaning up their product's build system as nonessential to their work and hence something that lower paid people should be doing?

Somehow, many developers have a mindset that non-feature work is not part of their job. This is not only preposterous but also extremely dangerous and is akin to a mechanic at a chemical plant believing the basic upkeep of his tools and workshop is not part of his job. If developers don't write automated tests, then chances are the culture of the company is focused on defect detection, not defect prevention. As shown in Chapter 5, this is the worst possible situation a software company can get itself into because it is extremely expensive and wasteful, with many defects being found by customers (which is the most expensive scenario of all). Likewise, if developers don't personally pay attention to the infrastructural details of their products (such as build and configuration management systems), it is all too easy for problems to creep in that eventually impact key performance indicators that also impact developer's productivity. Examples are build times, code bloat, source tree organization, and improper dependencies between build units creeping into the product.

How many managers refer to non-feature work as a tax that is unessential to product development?

If a company is in a feature war with its competition, the company's management needs to understand how the decisions and statements they make ultimately impact the long-term success of the company. Managers must recognize that they have a key role in leading the attitude of the organization. In this case, management, or even in many cases software developers themselves, need to realize that the only tax on the organization is their attitude about non-feature work. As in chemical manufacturing plants, it is counterintuitive that ongoing investment in continual improvement will lead to greater capability than solely concentrating on features and bug fixing.

How many teams are too busy to implement any improvements, such as adding to their automated tests, improving their build environment, installing a configuration management system, rewriting a particularly buggy or hard to maintain piece of code, because they're "too busy" developing features and bug fixing?

Just as the mechanics in the chemical engineering plants are unable to perform preventive maintenance by taking a pump offline on a regularly scheduled basis because they're too busy fixing problems (i.e., fighting fires), many software teams are also unable to pay attention to the "health" of the underlying software or the development infrastructure. As described earlier in this chapter, this is a key part of the software development death spiral. It will eventually lead to a virtual crisis with a team spending an increasing amount of time during each release fixing defects at the expense of other work. Either the company will eventually give up on the product or be forced into a likely rewrite of major portions of the software (or even the complete software) at great business expense.

When developers are fixing defects or adding new features, how often do they use shortcuts that compromise the underlying architecture because they're "under a tight deadline"?

The main danger with these shortcuts is that they're almost always commented (e.g., ugly hack introduced before shipping version 2) but rarely fixed. These shortcuts are broken windows (see chapter 4 for more details) in the software and are another part of the software death spiral.

Are the people who develop the products in your company proud of the products, but not proud of how the products were created?

If your products are absolutely brilliant but your people are burned out by the experience of individual releases, maybe it's time for a step back. Maybe there is too much heroism required, or too little planning takes place, or there is an ongoing requirement for long hours because the schedule is slipping. There are many possible factors, but the underlying cause can usually be traced back to a lack of focus on continual improvement and technical excellence.

The Genius of the AND versus the Tyranny of the OR

One way to think about continual improvement is through the genius of the AND and the tyranny of the OR [Collins and Porras 2002]. Software organizations need to stop thinking about features and bug fixing as being exclusive from underlying software health and infrastructure improvements. This is the tyranny of the OR; thinking you get features OR software health. By focusing on continual improvement through paying constant attention to sustainable practices, software companies will be able to achieve the genius of the AND: features and bug fixing AND underlying software health. Greater capability leads to an increased ability to introduce more features with fewer defects, and hence have more time to innovate, not less. Hence, focusing on continual improvement is not a tax it's an accelerator button^[3]!

^[3] Jim Highsmith deserves the credit for this observation.

A Sustainable Development Experience

I have been fortunate to work on at least one team that achieved sustainable development. In one case, I worked on a software project that was released to customers over the course of several years. Over this time, the capabilities, complexity, and size of our product increased but the number of defects in the product stayed constant and our ability to innovate increased. And our team did not increase in size. How did we do it?

• Our software worked every day.

• We relied heavily on automated testing to catch problems while we were working on new features.

• We had high standards of testing and code quality, and we held each other to those standards.

• We didn't overdesign our work and built only what our customers needed.

• We were uncompromising with defects, and made sure that all the known defects that were important to our customers were fixed in a feature before we moved on to the next one. Hence, we never had a defect backlog.

• We replanned our work as often as we could.

• We were always looking for ways to improve how we worked and our software.

Other teams in our company were incredulous that we could produce the amount of work we did and keep quality so high. For example, the company had an advanced practice of frequent integration, which was vital because the product was developed across many time zones. Because of our stability and quality we were able to pick up and fix integration problems extremely early in a development cycle. This was vital to the success of the company's products.

Think of Cobol when you think of sustainable development: The original developers of the Cobol language could not conceive of programs written in Cobol that would still be in use in 1999, and yet when the year 2000 came along, all of a sudden there was a huge demand to fix all the Cobol applications. Here's a related joke:

It is the year 2000 and a Cobol programmer has just finished verifying that the Y2K fixes he has made to a computer system critical to the U.S. government are correct. The president is so grateful that he tells the programmer that he can have anything he wants. After thinking about it for a while, the programmer replies that he would like to be frozen and reawakened at some point in the future so he can experience the future of mankind. His wish is granted.

Many years pass. When the programmer is woken up, people shake his hand and slap him on the back. He is led on a long parade, with people cheering him as he goes to a huge mansion. The master of the universe greets him enthusiastically. Pleased but puzzled, the programmer asks the master of the universe why he has received such a warm reception. The master of the universe replies "Well, it's the year 2999 and you have Cobol on your resume!"