Key Motivations for Iterative Development

Iterative development is lower risk; the waterfall is higher risk. Most practically, the motivation to adopt an iterative lifecycle rather than the waterfall is that research now shows the former is associated with lower risk and better success, productivity, and defect rates. It is these results that have led large and experienced software procurement organizations such as the USA Department of Defense (DoD) to promote the use of IID methods rather than the waterfall.

research

DoD evidence

Early risk mitigation and discovery. Risk-driven iterative development forces tackling the hardest, riskiest problems first, such as architecture, integration, and so on. And, early development iterations exercise and reveal the true nature of the team and individual skills, the tools, and third-party software. Finally, the truth of the risks emerges: Perceived risks prove not to be, and unsuspected issues are forced into the open.

Accommodates and provokes early change; consistent with new product development. IID methods work with rather than fight against the high-change nature of software projects.

Manageable complexity. Failure rates are higher and productivity lower with high complexity software projects. Iterative development decomposes complex projects or phases into small and bounded mini-projects of manageable complexity.

size research

Confidence and satisfaction from early, repeated success. Short iterations lead to a quick and repeating sense of completion, competency, and closure. These psychological factors are important for individual satisfaction and building team confidence. This also builds customer confidence in the team they see early visible progress in the direction they care about.

Early partial product. Not only does early visible progress with an integrated and tested partial product increase client confidence, it provides new business opportunities. Earlier demos are possible. And for whatever reason, the product can ship sooner with fewer features.

Relevant progress tracking; better predictability. Following the waterfall can give a false sense of progress during the early, easier phases, but with low reliability in predicting later phase schedules, which vary widely. A more meaningful progress indicator tested software is provided each iteration when using IID methods. That's Agile Principle 7. Further, since work in each iteration exercises most disciplines and each iteration is a similar mini-project, there is earlier and more representative progress data useful for future extrapolation and estimation.

tracking

agile principles

Higher quality; less defects. IID methods require testing early, often, and realistically, in all possible dimensions: load, performance, usability, and so forth. And the tests themselves can be evaluated and refined over the iterations.

defect research

Final product better matches true client desires. Through early evaluation and feedback from clients or prospective users, the product is more likely to hit the mark. This is a refinement of "higher quality."

Early and regular process improvement. A common practice in IID methods is a per-iteration assessment for example, a 15-minute discussion to discover a couple of concrete actions to take in the next iteration to address a problem or improve the living process. Broad-spectrum process improvement is enabled by IID, since work in many disciplines (programming, requirements, test, etc.) occurs each iteration.

Communication and engagement required. Failure research reveals that lack of client or end-user engagement is a major factor in software project failure. Likewise with lack of coordination and collaboration between members or sub-teams. Developing in iterations forces early integration, coordination, and communication between development team members. A per-iteration demo that requires the presence and feedback of clients increases their engagement, as does their participation in a per-iteration planning meeting in which they contribute to the choice of requirements for the next iteration.

failure research

IKIWISI required. There's a well-known human-nature related problem in software specifications, especially user-interface oriented: IKIWISI, or I'll Know It When I See It. The complexity, many degrees of freedom in solutions, and intangibility of software seem to demand concrete and cyclic feedback from people evaluating prototypes or partially built systems to clarify and refine their vision.

Timeboxing Benefits

Research shows that timeboxing itself brings benefits in terms of increased productivity. One reason is focus. Steve McConnell summed it up best in his book Rapid Development, "It's amazing how much you can get done the day before a vacation." The psychological focus promoted by a fixed end date only three weeks away is very different than if the team's next visible milestone is three months away. Timeboxing may be viewed as an antidote to Parkinson's Law: "Work expands so as to fill the time available for its completion" [Parkinson58].

Another value in timeboxing, both of iterations and of the entire project, is a quirk of human nature: People remember slipped dates, not slipped features. Delay a project three months from its original end date to include 100% of the desired feature set, and the "failure" will be remembered. Deliver on time with 75% of the most important features, and it's a success.

Another reason is being forced to tackle small levels of complexity. With a short two-week timeboxed iteration, the team takes on manageable complexity, gets realistic about what they can do, and has the ability to reduce the scope if it appears the deadline can't be met. Data shows that lower-complexity steps are done more productively.

productivity research

A more subtle benefit of timeboxing is its effect on early forcing of difficult decisions and trade-offs. For example, on a Scrum project, you have just committed to a 30-day timeboxed iteration. During the iteration planning meeting, the team has to be very realistic about what gets done and what gets deferred. Since the demo to the client is definitely in 30 days, there is no latitude to be fuzzy about the short-term goals and priorities. Stakeholders are forced to seriously consider priorities, early.