Although the original description is more iterative than many know (see p. 102), in common usage the waterfall or sequential lifecycle model meant the following development steps:
In the 1970s this was taught as the ideal approach to software development, in response to ad hoc code-and-fix programming in the 1960s. Many books, consulting companies, and teachers promoted the method as ideal, unaware of the accumulating evidence of its problems. Research now shows it is associated with higher risk, failure, and lower productivity.
The underlying reason for the difficulties of the waterfall is that it requires a low-change, low-novelty, and low-complexity problem. It is unsuitable for complex or inventive projects. Interestingly and unknown to many the author of a key waterfall paper (Winston Royce) said the idea was only applicable for the most straightforward un-novel projects; and most interesting, he was himself a proponent of iterative and evolutionary development.
The waterfall lifecycle pushes many high-risk and difficult elements toward the end of a project, while IID methods, run by risk-driven iterations, surface and resolve the hardest and riskiest elements early (Figure 5.4). For example, programming and testing the core architecture, integrating its major components and clarifying the interfaces, is tackled in the earliest iterations.
Figure 5.4. risk profile: waterfall
Figure 5.5. risk profile: iterative
The waterfall also aggravates complexity overload and analysis paralysis. Large steps with overwhelming degrees of complexity are attempted.
At some level, we need to know specifications before programming. On the scale of three or six weeks (an iteration or short project), the waterfall works. The breakdown occurs as complexity grows, change rates increase, and feedback is delayed; it works less and less well as a single step stretches from weeks to months to years.
As a historical footnote, in the 1970s and 1980s the DoD promoted the waterfall. In 1994, in response to failure and evidence of its unsuitability for software projects, the DoD changed its standards to remove the waterfall bias, and promote iterative and evolutionary development.
Senior executives within software organizations often received their software engineering and management education in the 1970s or 1980s, when the waterfall model was usually taught as "ideal." Therefore, it is understandable and common for some to be unaware of more recent evidence that this was not true.
Problem: "Complete" Up-front Specifications with Sign-off
It isn't that IID proponents wouldn't like correct and detailed up-front specifications that do not need to change if they could be successfully and efficiently created, very good. Yet, research shows it is rarely possible [CM95], and interestingly, a large study showed that 45% of features created from early specifications were never used with an additional 19% "rarely" used [Johnson02]. A different approach evolutionary requirements is needed.
Problem: Late Integration and Test
I recall working with a defense contractor in the mid-1990s; a large project failure had just wound down. The prime reason? After two years of distributed multi-site development involving over 200 developers, the last stage of the project was to integrate all the software components and do system testing. Characteristically, the integration effort overwhelmingly failed; fundamental misunderstandings and wrong assumptions in the communications, components, and how they would collaborate did not surface until this late stage. The problem was so intractable that the project was cancelled.
Problem: "Reliable" Up-front Schedules and Estimates
When adopting an iterative or agile method, at the start of a project before programming and the first iteration, do not attempt to create a reliable plan or schedule that lays out all the iterations, what will happen in each and when, or all the detailed activities sequenced in a PERT chart. Similarly, do not attempt (at the start) to reliably estimate the overall cost or effort, or milestones of intermediate completion points, or the end date.
Such early attempts at predictive planning are more successful on repetitive manufacturing projects of low change and complexity, but not on inventive projects where the full requirements and risks are not reliably known at the start, and high rates of change are the norm. As with up-front specifications, it isn't that IID practitioners wouldn't welcome reliable up-front estimates and schedules. But in domains of high change, complexity, and novelty, it is premature, risky, and unrealistic. Adaptive risk-driven or client-driven planning implies the scheduling of goals to iterations can and should change as better information and new priorities arise.
Still, the demand for premature up-front commitments happens all the time. This doesn't invalidate applying an iterative or agile approach it's just not ideal. Early estimates can be improved with the iterative estimation technique of Wideband Delphi. The iterative and agile practice of early visible progress and client-driven iterations can win the customer's confidence so that they eventually appreciate their dynamic steering control, and adopt adaptive planning, ceasing to care if the team is doing, week by week, what they were originally forced to speculate and schedule.
By developing in risk-driven iterations, the team will earlier discover the depth of their predicament and can react with mitigating actions, such as hiring specialists.
The waterfall has been called a fail-late lifecycle; one can have the illusion of an accurate schedule during the early, easier phases. This is because the hard and risky elements (like integration and test) are pushed towards the end. Then, halfway through the schedule as the real complexity and difficult elements surface pow! The schedule falls apart. It's like the story of the guy who fell off the cliff:
Problem: "Plan the Work, Work the Plan" Values
The old management maxim of "plan the work, work the plan" is suitable advice for predictable manufacturing. For high-change, novel, inventive domains such as software development, it has limited value except at the highest level of very coarse-grained activities or milestones. The maxim is associated with the values of waterfall development, predictive planning, command-control management, up-front specifications, and so forth. This is inconsistent with the agile method principles of adaptive planning, self-organizing and self-directed teams, and evolutionary development.
Of course, this does not imply that agile methods avoid planning or preparation. But the degree of detail, and commitment to plan are more light and flexible. Plus, the devolution of decision making and task assignment to the team itself rather than the manager changes the tone and goals of management planning.