Mistakes When Managing Iterative Development

Transitioning from a waterfall approach to an iterative approach requires changes in the working procedures of all team members , and it is sometimes especially difficult for project managers and architects . Organizations going through this transition often hold on to certain patterns and thought processes that lead to flawed project execution. Among these common mistakes, we find the following:

• Having a functional, specialized organization, as opposed to deploying cross-functional teams

• Not setting the right stakeholder expectations or using an old-fashioned acquisition model, making stakeholders expect the same deliverables and a project lifecycle as in a waterfall approach

• Having too many developers at project start, before understanding what to do

• Solving the easy stuff first, rather than first attacking high-risk items

• Having an extended initial iteration; trying to do too much at the beginning

• Having overlapping iterations; starting the next iteration before you can benefit from any feedback from the current one

• Allowing too many changes late in the project, leading to major delays, cost overruns, and potentially poor quality

Let's look at these mistakes in detail and examine some ways to avoid them.

Having a Functional, Specialized Organization

An iterative approach means that every iteration goes through a full lifecycle: analysis, design, implementation, integration, and test. If a project team is organized around these functional areas, then there needs to be a lot of knowledge transfer among these teams. This increases the risk of miscommunication (see Figure 13.1) and typically means that a lot more time is spent on unproductive activities such as documenting, managing, and reviewing everything. Sure, some documentation is good, but functionally oriented teams tend to spend too much time on documentation, taking away valuable time from productive activities such as developing executable software.

Figure 13.1. Functional Teams Have Inherent Communication Barriers. An iterative approach requires high-bandwidth communication, which is very difficult to achieve if analysts, developers, and testers work in different teams. Assemble a cross-functional team, including some generalists , to address this issue.

Ideally, the team should be cross-functional, with at least some generalists who have more than one area of expertise. In Inception, you typically start with a small team ”maybe 3 to 5 people for what may later become a 15-person project. For example, you might have

• A project manager who can help out with analysis.

• One or two analysts.

• An architect who can help to code the architecture.

• A lead developer to do design and implementation.

• A part-time tester for the whole system.

Each team member should also take responsibility for the quality of his or her own deliverables.

Not Setting the Right Stakeholder Expectations or Using an Old-Fashioned Acquisition Model

It is important to set the right stakeholder expectations. If executive management, customers, or other key stakeholders are used to another project lifecycle, you may experience a lot of problems. You may, for example, be required to

• Complete detailed planning of the entire project up-front, rather than creating an initial coarse project plan, complemented by detailed plans for the current and next iteration.

• Produce a complete requirements specification at the end of Inception, before funds are committed to the remainder of the project.

• Commit to a budget for the entire project by the end of Inception, before you have the necessary information. A majority of risks will be mitigated first during Elaboration, and any estimation of overall cost before major risks have been mitigated will by necessity contain a large degree of uncertainty.

These constraints may force you into following a waterfall approach ”rather than an iterative one ”with all the associated drawbacks. One common situation driving people toward a waterfall mentality is the use of an old-fashioned model for acquisition (contracting), where at the end of Inception you force a fixed bid on the remainder of the project, with fixed requirements, plans, and costs for the entire project. Rather, you should divide the project into two bids (see Figure 13.2), with one bid for the Elaboration phase and another (potentially fixed) bid on the Construction and Transition phases. This will drive the project toward a more economical and sound project lifecycle and avoid major risks such as late discovery of costs, schedule overruns, and failed implementations of requirements.

Figure 13.2. Break Up a Project into Multiple Bids. Subcontracting of iterative projects should preferably be done through two or more bids. As an example, you can have one bid for the Elaboration phase and another bid for the Construction and Transition phases.

Too Many Developers at Project Start

A couple of years ago, one of our colleagues came to us and complained about a project on which he had just started work. Management had put 40 developers into the project from the start, together with a group of 7 or 8 analysts and architects to analyze the requirements and decide on the architecture. Our colleague, who was the lead analyst, continuously had developers in his office asking him for tasks , forcing our colleague to spend more time making up bogus tasks for the developers than analyzing requirements. There is no point adding large numbers of developers to a project before you have a baselined architecture and before you know what problems you are trying to solve. ^[1] Everything that is built is being built on quicksand, you typically end up having to rewrite almost everything that is written, and the first delivered system becomes a patchwork .

^[1] There are naturally exceptions. In one project, we tasked a set of developers to independently develop a framework for later use, while we figured out what the application should do. This was somewhat successful since the developers happened to be very familiar with the domain and could make good assumptions of what components they would need later on. The risk with this approach, however, is that you may have to throw away a lot of your components that were not needed because they did not address the requirements.

Note that you need some development resources during Inception and especially in Elaboration. Since you need to design, implement, and test the architecture, you should typically have your most skilled developers involved during Elaboration, as they are building the skeleton structure upon which all future development is based. So have a small, highly skilled team through Elaboration, and scale up with additional developers as the architecture stabilizes (see Figure 13.3).

Figure 13.3. The Number of Developers Should Be Limited at Project Start. You want to avoid having too many people in your project at the start. This figure shows that you typically start with a small team in Inception, increase a little in Elaboration, and then add more developers and testers to the team during Construction.

Solving the Easy Stuff First

It is often tempting to say, "This is a delicate issue, a problem on which we need a lot of time to think. Let us postpone its resolution until later, which will give us more time to think about it." The project then embarks on all the easy tasks, never dedicating much attention to difficult problems. When it comes to the point at which a solution is needed, hasty solutions and decisions are made, or the project simply derails.

You want to do just the opposite : Tackle the hard stuff immediately. You should have a continuously updated risk list that should guide you in what to focus on in any given iteration. If at all possible, always attack the biggest risks first. If a project must fail for some reason, let it fail as soon as possible, before you have expended all your time and money.

Another related common mistake is not to continuously update the risk list. You performed a risk analysis at Inception and used it for planning, but then you forgot about risks that develop later in the project ”and they come back to hurt you later. Risks should be re-evaluated constantly, on a monthly, if not weekly, basis. The original list of risks you developed was just tentative. It is only when the team starts doing concrete development that they will discover many other risks.

Having an Extended Initial Iteration

There is a risk in trying to do too much in the first iteration. Typically, there are one (maybe two) iteration(s) in Inception, and you want to make sure that those iterations are successful. This means that you need to deliver what you said you would deliver ”on time. Sure, some minor delays might be acceptable, but massive delays are not. They set a bad precedent for the project team and make failure likely for the entire project. During Inception, you want to establish an even rhythm for the project: "We deliver a release every six weeks; we deliver a release every six weeks; we deliver ." Grady Booch refers to this as "developing a steady beat." ^[2]

^[2] See Booch 1996.

If you do start running very late, do not just let the first iteration go on forever. Cut the scope, and if necessary add one more iteration, but make the team commit to a certain iteration length ”two, four, or six weeks, for example. A steady rhythm will help you monitor whether the project is on track and distribute the stress in smaller increments every two, four, or six weeks, rather than subjecting the team to one big panic during the two months prior to delivery (see Figure 13.4). Note that the iteration length may vary based on objectives of the iteration and which phase you are in, but planning is, in general, simplified if iterations are of similar length (at least within the same phase). See Chapter 12 for more a detailed discussion on project planning.

Figure 13.4. Similar Iteration Lengths Help to Develop a Project Rhythm. It is important not to have the first iteration be too long. Ideally, each iteration within the same phase should be the same length, so that the project will develop an even rhythm.

Having Overlapping Iterations

Another very common trap is making iterations overlap too much (see Figure 13.5). Starting to plan the next iteration somewhere toward the last fifth of the current iteration, while attempting to have a significant overlap of activities (for example, starting detailed analysis, or designing and coding the next iteration before finishing the current one and learning from it), may look attractive when staring at a Gantt chart, but it will lead to problems. Some people will not be committed to following up and completing their own contribution to the current iteration. They may not be very responsive to fixing things, or they will just decide to take any and all feedback into consideration only in the next iteration. Some parts of the software will not be ready to support the work that has been pushed forward, and so on.

Figure 13.5. Large Iteration Overlap Defocuses Teams. Having a large overlap between iterations may make team members less committed to completing current tasks and instead more apt to jumping to solve the latest problem. Overlapping iterations can also radically increase management overhead and prevent an effective feedback loop.

Although it is possible to divert some personnel to perform work unrelated to the current iteration, this should be kept to a minimum and only as an exception. This problem is often triggered by the narrow range of skills of some of the organization's members or a very rigid organization: Joe is an analyst, and this is the only thing he can or wants to do; he does not want to participate in design, implementation, or test. Another negative example: A large command and control project has its iterations so overlapped that they are basically all running in parallel at some point, requiring management to split the entire staff across iterations with no hope of sending back lessons learned from the earlier iterations.

Allowing Too Many Changes Late in the Project

In Chapter 2, we discussed that you should embrace change, especially early in the project. Change is good; it allows you to improve your solution. But iterative development does not mean scrapping everything at each iteration. Scrap and rework have to diminish from iteration to iteration, especially after you have a baselined architecture at the end of the Elaboration phase. If you allow too much change too late in the project, you are likely to run into delays, cost overruns, and potentially low or unsatisfactory quality. This means that you need to carefully consider what changes are worth making, especially toward the end of the project.

Developers often want to take advantage of iterative development to do gold plating : to introduce yet a better technique, to rework, and so on. The project manager has to be vigilant so as to avoid the rework of elements that are not broken ”ones that are OK or good enough. Also, as the development team grows in size and as some people are moved around, new people are brought in. They might have their own ideas about how things should have been done.

Similarly, customers (or their representatives in the project: marketing or product management) may want to abuse the latitude offered by iterative development to accommodate changes or to change or add requirements with no end. This effect is sometimes called "requirements creep." Again, the project manager needs to be ruthless in making tradeoffs and in negotiating priorities. Roughly, by the end of the Elaboration phase, the requirements are baselined, and unless the schedule and budget are renegotiated, any change has a finite cost: Getting something in means pulling something out. Remember that "Perfect is the enemy of good."