Standards-Body Evidence

The transition of the USA DoD standards from waterfall to iterative and evolutionary is instructive. In 1980s a DoD standard for software development or procurement was released, DOD-STD-2167, based on a waterfall model and document-driven approach.^[2]

^[2] It also had an influence on standards set by other governments. The story is explored in the "The Historical Accident of Waterfall Validity?" section on page 102.

As mentioned earlier, the DoD was experiencing failure in the acquisition of software based on a waterfall approach and the 2167 standard. For example, a 1995 report on failure rates in a sample of earlier DoD projects drew grave conclusions: Out of a total cost of $37 billion USD for the sample set, 75% of the projects failed or were never used, and only 2% were used without extensive modification [Jarzombek99].

Thus, there was a motivation to improve the 2167 standard. In the latter 1980s, Firesmith, motivated from the stultifying influence of 2167 on the Magnavox project, helped lead the Ada community's promotion of IID to replace 2167 with a new IID-friendly 2167A standard. In collaboration with Lt. Colin Gylleat (who was deeply involved in producing 2167A), they were able to remove the legal requirements for functional decomposition and the waterfall development cycle. However, the original single-step waterfall diagrams remained in the updated 2167A from 2167, because (in the words of Firesmith):

Although the waterfall diagrams did not have any legal impact, I could not get them removed because the military logistics people would not agree with my assessment that they would continue to foster the waterfall mindset.

Magnavox project

His assessment would prove correct. Shortly thereafter, another push to replace the waterfall with evolutionary IID was made in a task force report [DSB87], chaired by Dr. Frederick Brooks. The report recommended to replace the waterfall, disproved on many large DoD projects, with iterative development:

DOD-STD-2167 likewise needs a radical overhaul to reflect modern best practice. Draft 2167A is a step, but it does not go nearly far enough. As drafted, it continues to reinforce exactly the document-driven, specify-then-build approach that lies at the heart of so many DoD software problems.

And on incremental acquisition and development:

In the decade since the waterfall model was developed, our discipline has come to recognize that [development] requires iteration between the designers and users.

Finally, in the section titled Professional Humility and Evolutionary Development (humility that it should be accepted that the waterfall goals of getting the specifications or design accurate without evolutionary implementation and feedback are rarely possible):

Experience with confidently specifying and painfully building mammoths has shown it to be simplest, safest, and even fastest to develop a complex software system by building a minimal version, putting it into actual use, and then adding functions [and other qualities] according to the priorities that emerge from actual use.

Evolutionary development is best technically, and it saves time and money.

The updated DOD-STD-2167A (Feb. 1988), which is often viewed by both DoD overseers and contractors as the epitome of a waterfall specification, was ironically actually an amendment promoted by Firesmith and Gylleat (and others) to allow lifecycle neutrality, to encourage IID alternatives to the waterfall:

This standard is not intended to specify or discourage the use of any particular software development method. The contractor is responsible for selecting software development methods (for example, rapid prototyping) that best support the achievement of contract requirements.

Despite this intent, the new standard was interpreted with justification by many to still contain an implied preference for the waterfall model, due to its document-driven milestone approach.

Due to ongoing failure with waterfall projects, and to re-emphasize replacing the waterfall with IID for DoD projects, a Report of the Defense Science Board Task Force on Acquiring Defense Software Commercially, June 1994, was issued that stated:

DoD must manage programs using iterative development.

Apply evolutionary development with rapid deployment of initial functional capability.

As a consequence, 2167A was superseded by MIL-STD-498 in December 1994. In "Changes from DOD-STD-2167A to MIL-STD-498," Crosstalk: The Journal of Defense Software Engineering, April 1995, Maj. George Newberry summarizes the changes to encourage "evolutionary acquisition" and IID in the section Removing the Waterfall Bias:

MIL-STD-498 describes … incremental builds. Each build implements a specified subset of the planned capabilities. The process steps are repeated for each build, and within each build, steps may be overlapping and iterative.

The MIL-498 standard explains and promotes evolutionary requirements and design; for example:

If a system is developed in multiple builds, its requirements may not be fully defined until the final build. … If a system is designed in multiple builds, its design may not be fully defined until the final build.

In 2000 the DoD 5000.2 acquisition "instruction" was released, that again recommended evolutionary delivery and the use of IID:

There are two approaches, evolutionary and single step [waterfall] to full capability. An evolutionary approach is preferred. … [In this] approach, the ultimate capability delivered to the user is divided into two or more blocks, with increasing increments of capability.

… software development shall follow an iterative spiral development process in which continually expanding software versions are based on learning from earlier development.

See also the scanned memo in Figure 6.6 that clearly communicates the DoD preference for evolutionary and "spiral" (iterative) development.

Figure 6.6. scanned memo

Thus, the DoD perhaps the world's largest and most experienced procurement agency for software started with the assumption that a waterfall model and up-front specifications was best (codified in 2167), and then based on the experience of high rates of project failures adopted iterative and evolutionary methods, demoting the waterfall.

The practice of IID is important to the DoD. For example, there is now an annual conference on the subject, Evolutionary Acquisition & Spiral Development, organized by the Institute for Defense and Government Advancement. See www.idga.org.

Unfortunately (for taxpayers worldwide) STD-2167 influenced the definition of standards in other countries, who did not keep up with the fact that the DoD later abandoned 2167 and the waterfall; many of these standard bodies still impose single-pass, document-driven processes.

The DoD is not the only standards group to make this shift. In 2002 the USA Food and Drug Administration (FDA) updated their prior waterfall model requirements [FDA97] for software development of FDA approved devices (e.g., medical devices) with a new standard that promotes iterative development [FDA02]. To quote,

Most software development models will be iterative. This is likely to result in several versions of both the software requirements specification and software design specification.

Although a number of European standards are still waterfall oriented, promising signs of change have emerged, such as the 2002 Bonn Germany NATO Symposium on Evolutionary Software Development.