This section summarizes some tips related to iteration timing, planning, tasks, tracking, and other project management practices to support iterative or agile development.
There is a trend in some of these tips: The manager is not alone in doing the work of planning, scheduling, estimating, tracking, and so forth. In agile development, it is more of a team sport.
Multiteam or Multisite Early Development
For projects that will be composed of multiple teams, perhaps spread across different locations, consider doing the early iterations at one location, in one common project room, with a small group ideally composed of one or two skilled representatives from each of the subteams. During these iterations, there is an emphasis on requirements analysis and development to discover and build the core architecture of the system the foundation. Major components, and their collaborations and interfaces are ideally clarified through early programming and testing rather than just speculative design. If UP is your method, this covers the inception and elaboration phase.
In this way, the early project benefits from the close communication, collaboration, common vision, and technical strength of the initial group.
Once the core is built, the representatives return to their respective locations, form larger teams, and the remaining work is done in parallel with multiple subteams. Each representative has developed a clearer picture of the vision and architecture, and can better convey and maintain that for the remainder of the project. Further, each acts as a liaison to the other teams. Also, after having spent some close time with the other subteam leaders, there is improved communication between the subteams. Figure 11.1 illustrates.
Figure 11.1. multiteam development
Difficult Multiteam or Multisite Iteration Planning
It is a worthy goal to have all components, across all subteams or sites, integrated and tested together to conclude the release of a common iteration. However, there are times when relaxing this goal is helpful. For example, consider the example of a multisite (five sites) research-oriented project spread across 16 time zones, working on a 3G telephony system that included creation of everything from the 3G handset to the protocol stack [Crocker02].
In this case, due to barriers in time, communication, and the difficult-to-predict research-oriented development, it was seemingly impossible to get all teams to coordinate and complete all work by a common iteration end date. For example, one team working on one layer of the system might need much longer than expected, or than the other teams, to complete the supporting goals of the iteration.
The successful solution was to establish the rule that at least two of the five teams had to define a common, relatively short iteration with integration between the two (or more) teams, and that no team could go "too long" without participating in a joint iteration. All the subteam project managers and technical leaders would adaptively plan the next iteration. In this way, the product elements were integrated in a series of disjoint iterations. Figure 11.2 illustrates.
Figure 11.2. sub-team iterations
This approach should be the exception rather than the rule. Although helpful for this multisite research-oriented project, the preferred guideline is that all teams work together towards a common goal and integrate on a common iteration end date.
Overlapping or "Pipelining" Activities Across Iterations
Some projects benefit from "pipelining" or overlapping certain activities across iterations, usually requirements analysis and testing [Larman97, JPKP03]. Figure 11.3 illustrates. For example, during iteration N, one or more people are doing requirements analysis for future development in iteration N+1, and a test team is evaluating the release of iteration N 1. In the latter case, defects discovered by the test team are handled in the current iteration if there is planned slack in the iteration schedule for this (and the defect is small), or deferred for a future iteration.
Figure 11.3. pipelining iteration work
Note that if there is a test team evaluating a prior release (probably for difficult or long-running tests), there should still be plenty of testing for the current release. The release should be as stable and fully tested as possible.
There may be efficiencies with pipelining, but not all projects can or should apply this approach. Furthermore, although the example shows pipelining of both requirements and test, it may of course be applied with only one activity. It is more appropriate for larger projects, those with offsite requirements donors, or those with long, complex testing that must be performed by a separate team. Examples of complex testing includes labor-intensive manual or semi-manual GUI testing, and memory-leak stress testing where the system needs to run for many hours or days to discover subtle leaks.
Pipelining has a seductive aura of efficiency when you look at a chart such as Figure 11.3, but as the agile methods stress, projects and people are not machines or computer processors. There are soft issues that carry potential pitfalls [Kruchten00a].
One pitfall is the slippery slope of overlap and focus. It may succeed for requirements and test, but can be part of a trend to a more general lack of current-iteration focus in other disciplines, such as programming. Working on past or future iteration "stuff" is not in the spirit of "let's work together on the next short step."
Another, related pitfall is a reduction in synergy, cross-discipline insight, "whole team together," "generalists over specialists," "customer close to developers," feedback, and adaptation all qualities promoted by the agile methods. Pipelining can re-introduce some of the deficiencies of a sequential waterfall model. It requires separation of people by discipline. For example, the developer may no longer be involved in requirements analysis (such as at a requirements workshop), yet research shows this is desirable [KC94]. In a degenerate case, if design work was also pipelined, the programmer would not be the designer. This latter situation is especially inconsistent with agile and adaptive development, where the design emerges from an interplay of some up-front speculation or design thought, combined with programming and testing to prove, disprove, and adapt the evolving system. The programmer and designer must be one and the same, or where design specialization is required such as in database or UI design the programmer and designer need to collaborate within the current iteration on common goals. In general, pipelining can reduce interaction and feedback between the disciplines and people; that's undesirable.
In conclusion, it's a double-edged sword.
Rolling Wave Adaptive versus Predictive Planning
One of the big ideas of agile and iterative development is to adapt based on feedback, and this is not only with respect to requirements and design, but also the plan or schedule.
This is the topic of adaptive planning versus predictive planning [Fowler01]. Adaptive planning in IID methods is a refinement of the well-known rolling wave planning concept more on this later. It implies that there isn't a detailed plan of all the future iterations. There is no fixed plan of how many iterations there will be, how long they are, or what will happen in each. In contrast, predictive planning implies there would be such an iteration-by-iteration plan to the project end.
This does not mean there are no large-grained milestones with dates, or no thinking ahead. In adaptive planning, there are milestones with dates, (or at least, there can be) but the path of iterations to those milestones is left flexible or adaptive. And, the milestones themselves may change if in the best interest of the project.
Rolling wave or adaptive planning is a key idea in agile methods. Comments on why this is a good idea in fact, a better approach than a detailed speculative schedule will come later, but first an example to help clarify the ideas:
Imagine that after two short exploratory iterations (that include programming and test) into a project, there is enough requirements clarification to be able to say with some certainty that the overall high-level requirements (e.g., use cases and features) are R1, R2, …, R20. Plus, there are rough effort estimates for each, and some understanding of their dependencies. Perhaps R2 must be done before R7, and so forth.
At this point, the customers indicate that they would like to see R1 through R10 by roughly mid-project, and the remainder by the end. The customers ask for an estimate of when R1 R10 can be finished. The customers may be asking for this milestone because of a demo they want to schedule for investors, a trade show, to synchronize with other product development, or simply because they are used to the idea of milestones.
The team does the best it can in estimating R1 R10, and believes it can finish in three months, on July 1, given an estimate of average developer availability. Likewise for R11 to R20, with a completion of September 1. Thus, two milestones have been established, and the team makes a commitment to meet these goals, with an understanding that change is possible later, in light of new insights or priorities. Figure 11.4 illustrates.
Figure 11.4. milestones estimated
Note that these milestones are more coarse grained than iterations; there may be many iterations up to the first milestone.
Predictive planning would go further than this, it would speculate a week-by-week or iteration-by-iteration schedule of the path to these milestones. For example, that there will be three iterations before milestone-1, each four weeks long, that R1 and R2 will be done in iteration-1, R3 R5 in iteration-2, and so forth. See Figure 11.5.
Figure 11.5. predictive plan
Rather, in adaptive planning and this is the key point we primarily plan in detail for just the next iteration. This is not an absolute rule. We may see some obvious dependent work to do in the follow-up iteration, or we may know that Jill the database expert will only be available in June, or that the new servers we ordered will arrive July 15. Such presence or absence of resources places obvious constraints on when we schedule certain tasks or requirements. But, all other things being equal, we primarily focus on deciding what to do in the next iteration, and defer making decisions about future iterations, unless obvious or necessary. See Figure 11.6.
Figure 11.6. adaptive plan
In agile and adaptive planning, the precision is commensurate with the information; the level of detail and commitment to fine-grained scheduling drop as we look further into the future.
Avoiding far-future predictive planning and preferring short-term adaptive planning is not a new or radical idea. It is also called rolling wave planning [Githens98], a well-known practice promoted in the PMBOK (Project Management Body of Knowledge) [PMI00], many management courses and texts, and government organizations. That said, there is often a gap between published management recommendations and actual practice or attitudes.
Government oversight agency: Oh yes, we promote rolling wave planning, as indicated in Standard-773-1. When will the weekly schedule and PERT chart for our new project be finished?
Regardless, the essence of rolling wave planning is to plan in detail only up to some realistic planning horizon, beyond which things are too speculative. In agile methods, that horizon is the next iteration.
What plan do external stakeholders see? If they want or need to establish milestones, they see a coarse-grained milestone-oriented schedule. A path of iterations to those milestones is not shown.
Finally, to re-emphasize points made before (since this is a common misunderstanding), adaptive planning does allow for coarse-grained milestones with dates, and can allow for some distant iteration plans when there are obvious resource constraints.
Benefits of Rolling Wave Adaptive Planning
Why is this useful? Adaptive planning is closer to optimal in terms of working towards milestones; each step can be the most skillful we know how to plan regarding risk, productivity, and effectiveness because each planning step is taken with maximum and fresh information. We take a step, and then ask, "Given what we now know, what is the most skillful thing we should do in the next step to work towards our milestone goal?" And repeat.
In contrast, a predictive plan is suboptimal. In fact, it could be close to the worst or most risky possible path to the milestone goal because it is created with the least amount of information, speculating into the far future. The schedule in Figure 11.5 could be a very poor path to the goals; we can't know. It is not wise or useful to believe that five iterations in the future, 17 weeks from today, the best thing we should do is implement requirement R15.
Not only is predictive planning suboptimal with respect to effectiveness and risk, it doesn't account for opportunities. If half-way into the project the marketing manager discovers our competitor is adding the sexy Gromlit feature, a predictive plan does not account for this. Yet, an adaptive plan does; the team can start adding Gromlits the very next iteration. Adaptive plans embrace change and opportunity; predictive plans fight or ignore it.
In addition to supporting adaptation, another key, basic reason for not creating an early detailed schedule is that in evolutionary methods not all the requirement and design details are known near the start of the project. Two weeks after project initiation, you can't plan for use case X in iteration-7 when you don't even know about use case X.
Although rolling wave or adaptive planning has been promoted for years in leading management circles, there are still some inconsistent software project management texts and courses that teach it is ideal to create a detailed predictive schedule or PERT chart of week-by-week activities through to the end of a software project. And, that failing to follow it is a sign of lack of skill on the part of a manager. Ironically, deviating from a speculative plan in response to risks or opportunities is a sensible response, not a failure.
As with the waterfall model, the heart of the problem with predictive planning is that it is misapplied to software projects. It works for mobile phone manufacturing; software is a domain of new product development, not predictable manufacturing.
Planning: Consider Ending on "Wednesday"
If wrapping up an iteration is nontrivial (for example, on larger projects with several subteams), consider making the last Wednesday (or Thursday) rather than Friday the goal date for baselining the release. Although the goal is definitely Wednesday, allow Thursday and Friday as slack days for unanticipated problems.
Thursday and Friday also provide time for group planning of the next iteration, perhaps another requirements workshop, a group iteration assessment or review, a demo to stakeholders, and so forth.
Further, if there is a separate test team that receives the end-of-iteration release (pipeline testing the release of iteration N-1 during iteration N), make Thursday the day when that team expects to start [Blaustein03], again allowing two slack days. This approach decreases the likelihood of working over the weekend to meet a Monday start-of-iteration hard deadline, when everyone is expected to move forward on a new iteration rather than still be working to wrap up the release to hand over to the test team.
Planning: Whole Team Planning Meetings
Agile methods emphasize collaborative planning. An initial release planning meeting and each iteration planning meeting are ideally held with all developers and customers present. If the project is large and composed of subteams, at least include representatives from each subteam.
Planning: Workers Estimate
The agile methods promote the value that the workers doing the work must estimate the work. At the planning meetings, involve the developers in estimation of large-grained requirements that will be allocated to iterations, and also estimation of the related finer-grained tasks. In XP, for example, it is required that the worker who volunteers for a task must be the one who estimates it.
Planning: Improving Estimates with Wideband Delphi
Estimation is a large topic mostly beyond the scope of this introduction. However, in addition to the agile methods practice of workers estimating their tasks, it is worth considering the technique of Wideband Delphi for large or valuable projects [Wiegers00]. This is an iterative, adaptive method (hence its inclusion in this chapter) complementary to other estimation methods. The practice is common in successful outsourcing companies whose estimates have low variance to final actuals.
A key point is that Wideband Delphi is used to complement an existing estimation method, not replace it. Here are the steps:
Note that Wideband Delphi sits on top of any other estimation method, improving it through multiple participants, feedback, and iterative refinement.
Planning: Multiple Iteration Planning Meetings
As discussed, agile methods apply adaptive planning; thus, at each iteration (usually near its end), there is an iteration planning meeting for the subsequent iteration, during which it is decided what to do in the next, followed by detailed task generation.
Planning: Agile Task Generation
During each iteration planning meeting, the team generates finer-grained tasks (for example, in the half-day to two-day range) for the iteration requests. XP promotes the approach of either having the team work together to write these on a whiteboard, or each team member brainstorms tasks and writes them on cards, that are grouped and stuck on a wall. In other words, an "agile" approach to a work breakdown structure is performed.
Re-use is worthwhile here; many tasks repeat across iterations and projects. If the final task list is only handwritten on a wall, consider taking a photo of each version, and share these for inspiration at future planning meetings.
Planning: Don't Forget to Budget for Iteration Overhead Tasks
More frequent risk management, iteration demos, iteration planning meetings, a group iteration assessment, a daily Scrum meeting, some average rate of unexpected defects, infrastructure failures, and so on, all consume time, and need to be reflected in the iteration task list. Managers new to iterative development sometimes miss accounting for these overhead tasks.
Planning: People Estimate Their Time Budget Each Iteration
Timeboxing requires a realistic approach to time management. People have to avoid overcommitting, as there is not much slack. So, it is helpful to calculate personal time budgets also an XP practice. During each iteration planning, people estimate their total "ideal work hours" or uninterrupted attentive time for project tasks for the iteration. This analysis may take five or ten minutes. A common average is around five hours per day if it is much higher, be skeptical.
A simple practice related to commitment, motivation, and job satisfaction is to promote volunteering for tasks rather than task assignment by a manager. This approach is part of XP and Scrum.
During the iteration planning meeting, after task generation, people volunteer for enough tasks to get busy. As the days pass and they finish tasks, they volunteer for more. Hold each daily Scrum by the "task list" wall, and the team has the basic information necessary to volunteer.
What if no one volunteers for a particular task? Rather than answer the question prescriptively, it is instructive to learn the agile project management attitude in a method such as Scrum. In Scrum one would respond to this problem by saying, "It isn't the project manager's responsibility to solve this, although she can offer advice or resources in response to a request. We work by self-directed teams, not manager guided. It is the team's collective problem to solve it." This level of self-direction and decentralization of responsibility is a significant value change for some managers or organizations.
Visible Project Plans
Where possible, prefer to show all iteration tasks on a wall or whiteboard. If a daily Scrum meeting is practiced, hold the meeting by this wall. During the meeting, when tasks are reported complete or added, they can be crossed out or written in; likewise with the project blocks or impediments.
Iteration Goals: Risk, Coverage, Criticality, Skills Development
What should be done in the earliest iterations? Rank requests and iterations by risk, coverage, and criticality.
Risk includes both technical complexity and other factors such as uncertainty of effort, poor specification, political problems, the need for a novel UI look and feel, or usability.
Coverage implies that all major parts of the system are at least touched on in early iterations perhaps a "wide and shallow" implementation across many components. The goal is to discover and stabilize the major software and hardware components, their interfaces, and collaborations. That's an important part of the overall architecture. For example, early use case scenarios may be chosen that require execution (and thus development) across many of the components (e.g., from UI to database). These are called architecturally significant use cases.
Criticality refers to functions of high business value; these need at least partial implementations in the earlier iterations, even if not technically risky. This driver may be considered a kind of political or business risk: if the paying customer does not see early evidence of things they care about, their confidence or sense of collaboration with the project team drops. Early inclusion of marketplace-relevant features also makes it easier to demo the product and garner attention.
On some projects, another early driver is skills development one goal is to help the team master new skills such as adopting an agile method or object technologies. On such projects, skills development is a heavily weighted factor that tends to re-organize the iterations into less risky or simpler requirements in early iterations, motivated by learning rather than risk reduction goals.
Iteration Goals: What to Rank?
Use case scenarios, features, defects, and nonfunctional requirements (e.g., internationalization) can all be ranked with the above criteria of risk, coverage, and so on.
Use cases are often composed of many scenarios. For example, the "Process Sale" use case has one scenario involving paying by credit, and another for paying by cash. Some of these use cases are too complex to implement all scenarios within one reasonably short iteration; thus, they need to be decomposed by scenario for both ranking and scheduling.
Therefore, include all types in a ranking list.
Iteration Goals: How to Rank? Dot Voting
Based on the drivers, requests are ranked. The ranking may be informal and qualitative, generated in a group meeting by members mindful of these drivers. This approach is in the spirit of agile modeling.
Consider using the rapid scoring method of dot voting. List the requests on a whiteboard, or display them from a computer projector on a whiteboard. Everyone gets a whiteboard marker pen with which they can make 20 dots (for example). As a group, and in silence (to reduce influence), all approach the board and apply dots beside the items, reflecting the voter's priorities. A voter can assign many dots to one item, or distribute them. On completion, sort and discuss.
That may be sufficient. A slight refinement is to do a second round of silent dot voting to reflect updated insight based on first round voting and discussion. This second round provides the feedback and adaptation by which decisions improve.
The requirements ranking will be done before iteration 1, but then again before iteration 2, and so forth. In this way, the scheduling of iterations adapts to current information and opinion.
Iteration Goals: How to Rank? Quantitative Methods
Group discussion and something like dot voting for request ranking are often sufficient a collaborative, fast, and fuzzy approach very much in the agile spirit. For the more quantitatively minded, variations on the following have been used. The example values and weights are only suggestive; the point is that numeric values, weights, and weighted sums can be used to reason about priorities.
The exact values should not be taken too seriously, and note that the numbers don't tell the whole story. Even though logging is a low-risk, simple feature, it is architecturally significant because it needs to be integrated throughout the code from the start. It would be awkward and would diminish architectural integrity to add it as an afterthought.
Iteration Goals: Related Iteration Length?
Viewing an iteration like any project, the choices boil down to fixing time and filling it with work, or fixing work and filling it with time. The specific approach varies in some IID methods (such as the XP Planning Game), but general outlines follow.
Iteration Length Chosen First
First, comments on factors that affect iteration length:
Requests Chosen First
Either approach is best done in a collaborative planning effort with all team members, or subteam representatives for large projects.
Iteration Goals: Before the First Development Iteration
I've seen a project team start into iteration one, and quickly falter because the team is impeded waiting for the technical environment to get properly established and debugged: The source control management system, the continuous integration build machine, the application server, project Wiki, and so forth. Strive to have this in place beforehand; it always takes longer than expected.
Iteration Goals: The First Development Iteration
Or, what not to do. Iteration one proceeds slower than expected. So, use generous estimates, and choose small goals among the set of high-ranking requests. Then it is more likely that the team will successfully complete on time with less struggle, building confidence.
If the project also includes skill transfer goals (learning about object technologies, or a new agile method), do not dump a big bucket of new ideas and practices into iteration one. Add them incrementally over two or three iterations. For example, if the team has not used version control before, perhaps leave it and the related continuous integration practice for iteration two, and just emphasize test-first development and daily Scrum meetings in iteration one.
Iteration Goals: Use Cases and Scenarios
If use cases are being employed, it is desirable to fully complete a use case (or many use cases) within one iteration. Planning and scheduling are straightforward, and the growing system has functional cohesion.
This is not always practical. As mentioned in some other tips, use cases are composed of many scenarios (related to "extensions" in Cockburn's popular use case terminology). For example, the Process Sale use case has a scenario involving paying by credit, and another for paying by cash. Some use cases are too complex to implement all scenarios within one reasonably short iteration. Perhaps it would take three months to complete all scenarios of Process Sale undesirably long for an iteration. Thus, they need to be decomposed by scenario for ranking, scheduling, and tracking. In this case, an iteration should complete the scenarios that it starts; a scenario should not be split across iterations (Figure 11.8).
Figure 11.8. scenarios across iterations
Iteration Goals: Primary and Secondary Requests
By frequently tracking remaining effort estimates, it may eventually appear that not all requests will be completed within the iteration. In timeboxed methods the response is to remove or simplify requests, rather than extend the iteration. What to remove? On projects with a strong customer-driven emphasis (such as XP projects), consider this approach: During the iteration planning meeting with customers, after the requests have been chosen, classify some as secondary that may be deferred. Not only does this support the agile spirit of customer-driven work, it also aids expectation management.
Iteration Goals: Don't Add Requests to an Iteration
An important rule in timeboxed methods is to not allow the addition of new requests once the iteration is underway. The product manager can't come over and ask to squeeze in the sexy Gromlit feature. Gromlits have to wait for the next iteration. Given that an average iteration is two or three weeks, this should not be a hardship. This rule creates an island of sanity and control in a sea of change and chaos. Newly discovered tasks related to existing requests can be added by the team members, but not new requests.
Tracking Iteration Progress
Some methods, such as XP and Scrum, have specific tracking practices discussed in their respective chapters. Some general iterative and agile-oriented principles and practices include:
Tracking Iteration Progress What to Track?
Since timeboxing creates a fixed soon-coming deadline, the critical progress data are the remaining estimated effort on unfinished tasks. This is the focus of a Scrum Sprint Backlog.
If a project also collects actual hours spent on tasks for each worker, consider using a daily tracker to collect this. One immediately relevant use for this data is to share it with the team members during subsequent iteration planning meetings so they can compare their original task estimates with final actuals. This feedback helps people become more accurate estimators a valuable skill.
Test-driven development is an increasingly popular and valued practice in IID methods, not only in XP. To promote the early creation of more unit tests, some agile projects track (and update on a wall chart) the total number of unit test classes and test methods. The counting can and should be automated with a simple scanning program that runs as part of the continuous integration build process.
Tracking and Planning: XPlanner
If the simple "low-tech, high-touch" tools preferred in agile methods such as a whiteboard task list, paper task cards, or the Sprint Backlog spreadsheet file are found to be insufficient (more likely on larger projects), there are (free) open source software tools emerging to support agile planning and tracking. An example (not an endorsement) is XPlanner, available at www.xplanner.org. It provides support for stories, tasks, time tracking, and metrics.
Earned Value Tracking on IID Projects
Earned value (EV) tracking is a cost and schedule progress measurement method required on many USA government projects. Thus, it must be considered on government projects that want to apply an IID method. The details are beyond the scope of this introduction, but a non-obvious key concept is to measure progress in terms of the estimates (or budgets), not only in terms of actuals. For example, if creating help Web pages for a new system was originally budgeted at 50 person-hours, then when the work is complete, regardless of the actual time spent (for example, 80 hours), the project is said to have "earned" 50 hours of value. A key term in this context is the Budgeted Cost for Work Scheduled (BCWS) the estimates for future tasks, such as 50 person-hours for the help system.
There are a few practices experimenting with EV tracking on iterative and evolutionary projects, although their true worth is not yet known.
One practice is to re-calculate the BCWS (estimate) values each iteration, as more information arises. This has been called a rubber baseline.
Another practice is to apply a simple earned value recognition rule to iteration tasks. There are alternative rules, but a common one is that as soon as an iteration task is underway, it earns 50% of its value, in terms of progress tracking. Thus, as soon as the help page work starts, the iteration "earns" 25 hours of progress. Progress remains at 50% until the task is complete.
IID methods tend to be risk driven. A fast, useful method to prioritize risks is to estimate their probability and impact (in cost, time, or effort). The estimates may be quantitative (which are usually very speculative) or simply qualitative (for example, high-medium-low, based on discussion and group dot voting).
The worst risks requiring proactive actions are naturally those both probable and of high impact.
Agile modeling promotes the practice of visible models, and that is especially useful for a risk list, with associated actions and status. Display this information using whiteboards or posters on the wall of the project room.
Many iterative methods are risk driven, which includes not only tackling higher-risk technical elements in early iterations, but also more broadly, identifying and proactively working on all risks.
A key to successfully risk management is proactive actions owned by individuals, that are tracked. Consider keeping owner and status information on the project room's wall display of the risk list (see Table 11.1).