7.6 Principles of the Holistic IterativeIncremental Software Lifecycle

7.6 Principles of the Holistic Iterative/Incremental Software Lifecycle

The HI/I lifecycle is guided by a set of general principles. In this section, we'll look at each principle and explain how it relates to the goal of effective management.

• Manage requirements not tasks .

• The important goals are the business goals ”dates and budgets .

• Think like a businessperson ”What have you done for me lately?

• Divide and conquer.

• Cut the job into programs and projects (cut at the thinnest part).

• Tie everything back to the business.

• Create demonstrable deliverables.

• Learn the art of "good enough" quality.

• The pieces will be smaller than you think.

• Expect negotiation, not specification.

• Forget about baselines, sign-offs.

• Estimate by doing.

• Calculate return-on-investment in a new way using portfolios.

7.6.1 Manage Requirements Not Tasks

In a HI/I lifecycle, project managers are far more aware of the requirements of the project than we would be in a waterfall lifecycle. This is because the requirements are the building blocks for project scheduling and team assignments. For instance, in a waterfall world, a project manager might give a team the responsibility to "design the credit rating calculation component." This is a specific set of tasks that relate to design, not analysis, not coding, not testing, and so on. It is also scoped by stating that the team should attack only the credit rating calculation part of the business problem, not customer service, not interest rate calculation, not monthly statementing, and so on.

However, the interfaces between this activity and other activities are huge. Being designers, they will need to find out what requirements and analysis were done and understand that effort fully. Maybe they did those tasks, maybe they didn't. Then they will need to formulate the results of their design efforts in a way that will be easily understood by the development, testing, and deployment teams coming later. There are many, many, many interfaces between teams in this scenario. Not to mention the interfaces between credit rating calculation and the other business functions.

In a HI/I lifecycle, the assignment is quite different. A team will be assigned responsibility for implementing a specific set of requirements. Using the process we've described in this book, the team may be assigned implementation responsibility for a set of use case paths or nonfunctional requirements. For instance, the team may be assigned implementation responsibility for the use case path "Enter New Credit Card Customer ”basic path." The team will need to make sure the pieces are in place to make that functionality demonstrable by the end of the iteration. When they are done with the iteration, someone should be able to sit down at the computer and go through all the steps that are specified in the use case "Enter New Credit Card Customer" in the basic path . The user interface might not be very jazzy, error processing might not be working (that will come when we do exception paths), but the whole process is possible to complete from a businessperson's perspective.

This type of planning is very different from waterfall task assignment. But the advantages are many:

• The businessperson can see tangible progress at the end of each quick iteration (1 “3 weeks).

• The team has the ability to adapt when changes occur, even in the middle of the iteration, because the design has not already been set in stone.

• The team's motivation level starts and remains high because they are producing tangible results regularly, giving the "instant gratification" that so many people crave.

• The interfaces between teams are fewer than with waterfall task assignment. The team accomplishing this use case interfaces mainly with the architecture team, who ensures that the team's work fits into the overall technical structure, and with the business analysts, who ensure that the team's work fits in with the business goals.

• The team is composed of a cross-functional set of individual contributors: presentation designer/coders, business layer designer/coders, data access designer/coders, data modelers/database administrators, and other technical specialists. This helps create less animosity between disciplines (the testing group dislikes the development group , for example) because the project teams are judged on their success together, not as functional silos . Also, it allows for more variation of methods and practices, so experimentation can bring new ways of making the product better quality or produced more quickly.

7.6.2 The Important Goals Are the Business Goals ”Dates and Budgets

If you live in a community where a stadium has recently been built, think back to that lifecycle. The pronouncement came that the city would build a grand monument to itself, which would attract professional sports teams, circuses, and other incredible events. Various contractors got involved to create the stadium concept, an interesting architectural design, and a set of cost estimates. The project proceeded and finally was usable.

What do you remember from the stadium construction? Over budget? Behind schedule? Or on-time , within budget, with a few things undone?

What do you remember from the stadium lifecycle? If the stadium went over budget, you heard about it. The newspapers reported it, especially if the taxpayers bore the cost of the overrun , which is often the case. Also, if the stadium was late in coming to completion, everyone remembers that too. If the stadium was over budget or behind schedule, we noticed.

What is interesting is what we do not remember about the stadium. If a stadium was within budget and within schedule, but still had some components not operational at the time of "completion," did anyone talk about it? Let's say the stadium was declared complete, but several of the "nosebleed sections" did not have chairs put in yet. Did anyone care? Or, more accurately, did anyone care as much about that as they did about schedule and budget?

What was your role in the stadium lifecycle? You were probably a "user" if you were interested in using the stadium after it was built and also a "stakeholder" if you were paying the taxes to get it built.

Now, take that mind-set to the software projects you've been involved in. Which projects do you consider successes and which ones failures? Examine your own viewpoints here versus the stadium and see what is different.

In the software world, we have based our scoping restrictions on the feature set of the applications, rather than the schedule and budget. Get those requirements nailed down early, we think, and you'll have success. By defining the set of requirements early and then trying to discourage any changes to them throughout the lifecycle, we may be missing the business point. The requirements set should always be the most flexible in changes, while schedule and budget must be the least flexible. With the waterfall lifecycle, we've got it backwards , think our businesspeople. They're right!

7.6.3 Think Like a Businessperson ”What Have You Done for Me Lately?

A complaint that businesspeople often make is that the stock market has a short memory and little foresight. The focus is on today. Or, at best, "What have you done for me lately?" In fact, this is a reality of business, whether it is a public company or not. Business is unforgiving and temporal. Businesspeople are the same way, if they are successful. Most businesses are evaluated, internally and externally, by fiscal quarter (three months). Financial results are reported to the stakeholders and problems are discussed. Investments are evaluated for their contributions to revenue and profits.

Have you ever been on a software project that was cancelled? It may have been cancelled for "budget reasons." The executives were doing their quarterly evaluation of their investments, including the money they were pouring into your project, and they decided it was no longer worth it. "Cut our losses." Unfair? Probably. Why did it happen?

Projects that create results that tangibly help the business increase revenue, increase profit, or decrease costs are projects that are less likely to get cancelled. And how often do those results need to be created? At least quarterly. If your project has gone for more than three months without producing a result that helps the business, it is at risk. Right now, businesspeople are looking at that outflow of money, seeing no return yet and asking that question, "What have you done for me lately?"

7.6.4 Divide and Conquer

Of course, a project manager needs to take a large endeavor and break it up into smaller pieces. This is a fundamental task of project management. Work breakdown structure, task definition, and assignment, resource allocation ”this activity goes by many names . Essentially, there is a lot of work to do. The project manager needs to take that work and divide it up among a team of people to get it done.

The most important part of this activity is ”where to cut? What is the most effective way to carve up the big chunk of work and allocate it to teams who have different skills, skill levels, specializations, career paths, and tools? There are several common ways of allocation.

One is by technology . Certain people have certain technical skills, so you give them the work that matches what they are good at. For instance, one team may be responsible for presentation (user interface, Web pages, navigation, screen widgets, and so on). Another team may take the business rules layer (creation of the business objects, controllers, and so on). A third team may take the data mapping layer (moving data back and forth between the business rules layer and the relational database). A fourth team may test; a fifth team may create the database; a sixth may handle architecture, and so on. Each of these layers requires specific technical skills and experience with the architecture to get the work done efficiently .

Another is by subsystem . Business applications can typically be broken up into subsystems, handling work done by different business departments. In a human resources management system, the subsystems may be payroll, benefits, career planning, recruiting, and company social activities. Each of those subsystems could be assigned to a team who builds the components to handle that business function.

A third is by requirement (use case) . The teams are assigned sets of use cases and they are responsible for creating working software that makes those use cases possible.

Edward de Bono (de Bono 1994) has a creativity tool called PMI ”plus, minus, interesting. We'll use that tool here to compare these three methods of dividing and conquering.

7.6.4.1 Dividing and Conquering by Technology

Plus

• Teams are composed of like-skilled members who may systematize their technical disciplines and advance their approaches together (for example, testers together in a group may create a testing methodology).

• When you create a process where the baton moves from technical group to group, it is easy to see what the status is (for example, the process moves from the business analysts to systems analysts to developers to testers, and so on).

Minus

• Each group starts to see the other groups as their enemies (for example, those developers are always passing the code to us testers too late for us to do our job).

• Dividing by technology tends to reinforce the waterfall lifecycle (for minuses of waterfall, see Table 7.2).

• The interfaces between the groups are not clean. All groups need to pass a tremendous amount of information back and forth, especially if an iterative process is used (for example, groups will be attending each other's meetings, need to know each other's processes, and so on).

Interesting

What happens when technology changes dramatically for one group that has an effect on another group?

7.6.4.2 Dividing and Conquering by Subsystem

Plus

• This approach gives the team more appreciation for the business problems being solved , as each group specializes in part of the business.

• The technical groups tend to form better relationships with the SMEs in the business groups.

Minus

Business subsystems are often harder to determine than at first glance. If they are divided by business department, often there are situations where several departments need to use the same functions, so it becomes unclear which subsystem group should take responsibility. Also, with workflows, it becomes unclear where along the workflow one subsystem stops and the next one takes over.

Interesting

What happens when the departments are reorganized or reengineered? Will the system still function in a sensible way for the business?

7.6.4.3 Dividing and Conquering by Use Case

Plus

• Each piece, when developed, can be tested and demonstrated to the businesspeople as a coherent bit of functionality that accomplishes a business goal.

• Use cases are more fine-grained than subsystems or technology, so the smaller use case work packages can be shuffled around as teams need more or less work throughout the lifecycle.

• Each use case goes end-to-end, through presentation-business rules-data mapping-database-architecture, so everything can be integrated and tested early and often.

• Since the teams to accomplish one use case must be interdisciplinary, the teams become more cohesive and less likely to build walls between technical specialties (presentation, database admin, and so on).

Minus

• Project managers, who have been occupied with task tracking, now must understand the use cases and they will have to spend more time learning about the requirements.

• Can lead to a functionality-centric viewpoint, where nonfunctional requirements and systemwide architecture get ignored.

• If the architecture group is not strong, the use cases may be too independent of each other, where common components are developed multiple times and common patterns and metaphors are not followed.

Interesting

Will teams who are new to use cases be able to divide and conquer this way?

Because of the issues that occur with the other methods, we strongly recommend dividing and conquering by use cases. It results in better products, earlier results, more cohesive teams, and better architecture.

7.6.5 Cut the Job into Programs and Projects

In many software organizations, new groups have sprung up called Program Management Offices (PMOs). The idea of these groups is to centralize people who have the skill to manage large projects, a very rare skill indeed. Few jobs are as stressful or demanding as the job of a program/project manager.

It is worth defining the difference between project and program.

• A project is an undertaking requiring concerted effort.

• A program is a set of projects that benefit from being managed together.

A project manager takes responsibility for the execution of the "undertaking." He tracks the progress and handles the problems that arise. Our friend Jeff Jamison, a great project manager, says that the project manager's job is to "be further up the road from his team, clearing roadblocks and plugging potholes." He's right. The project manager has a vision of the upcoming weeks and months and is making sure that things fit together as the team progresses toward its goals and deadlines.

A program manager has a slightly different role. Since a program manager is responsible for multiple projects, she will be in contact with multiple project managers. Program managers are concerned with the interconnections between those projects: interdependent deadlines, functionality linkage, staffing swaps, requirements overlaps, and so on.

Program managers also get involved in reporting on the combined status of the multiple projects, including budgeting.

The role of the program manager versus the project team is illustrated with a metaphor provided to us by Jim Hendrickson, a friend, project manager, visionary , songwriter, and United States veteran:

Imagine that the project is a jet fighter that needs to land on an aircraft carrier in the South Pacific. The roles are

• Project manager (jet pilot)

• Program manager (air traffic controller on the aircraft carrier)

When the plane is farther than 50 miles away from the aircraft carrier, the pilot has a lot of freedom. He can fly around, do flips , do whatever.

Jet Landing on an Aircraft Carrier

Once he gets within 50 miles of the aircraft carrier, he must begin communicating to the air traffic controller (ATC). They negotiate the terms of the landing. The ATC instructs the pilot of his vicinity to other planes, as well as the need to fit in with other planes landing soon. The ATC also requires that the pilot decide on one of several available approach patterns. The pilot chooses an approach and continues moving toward the aircraft carrier.

As the jet gets closer to the aircraft carrier, the communication between the pilot and the ATC becomes more intense . The pilot is constantly stating his perspective: the readings from the gauges in the plane, his view through his windows , his requirements. The ATC is also giving her perspective: other planes that have landed and are moving out of the way, weather conditions, radar readings of other aircraft in the area.

During landing, the communication must be flawless ”back-and-forth messages about both perspectives, move up a little, your left wing is too low, and so on.

The communication starts fairly casually, gets more intense, and becomes ultimately meticulous and thorough through landing.

Note that the pilot (project manager) is at the controls of the jet throughout. At no time does the ATC (program manager) say "you're not doing well enough landing this plane, we're taking over." What a disaster that would be. Probably possible with wireless remote computer control, but not desirable by anyone. The same should be true of the relationship between program and project managers.

We have held back from saying that project managers "report to" program managers. We don't think that is exactly true. The program managers provide a service to project managers, but the project managers need to be ultimately in charge of "landing the plane."

Also, you'll notice something interesting with Jim's metaphor. The amount of communication starts small, then increases to a frenzy by the time landing (implementation) occurs. This is how the relationship should work on software projects too. Program managers need to give the projects a lot of latitude at the beginning of the project, but inch down on the specifics further and further as the team approaches implementation. This doesn't mean lazy or slack communications ”they still need to be precise ”but there can be less communication at the beginning than later on.

The Project Management Institute (PMI)

The Project Management Institute (PMI) provides a tremendous amount of literature, guidance, and training for project and program managers. They offer certification for a manager to become a Project Management Professional (PMP), moving the profession of project management a little closer to the professions of accounting, law, medicine, and so on. Being a true project management professional requires a tremendous amount of skill, a lot of courage, and excellent people skills, especially communication. PMI training for software development managers still focuses mainly on the waterfall lifecycle, but they are moving to correct that issue. (Web site: www.pmi.org)

Program management is a difficult concept to implement well. Certainly, you have heard of the Program Management Office (PMO), a group created to manage multiple projects in an efficient, effective manner. It is very hard to get PMO right, from what we've seen with our clients . The PMO can rapidly deteriorate into a glorified status reporting organization that does not provide much value to the project managers/pilots. Or the program manager can become a type of super-project manager, who manages the managers below him the way a project manager manages the individual project. This is not the intent of setting up programs, nor is it anything more than an annoyance for the project managers. Program managers must keep the role of the air traffic controller in mind with their projects.

Also, in a use case driven lifecycle, where divide and conquer by use case is used, there is another interesting wrinkle. Project managers manage more by requirement, less by task. This is a major change from waterfall project management. The tasks are driven from the use cases the team tackles. The project manager is more interested in "which use cases the team is working on" than the individual tasks people are accomplishing. This can allow the team to self-manage to the point that they can still accomplish the use case functionality in the time allowed.

How to Slice a Use Case

Is it really always that simple? No, never. Here are the issues that arise. Use cases usually contain too much functionality for a team of three or four people to accomplish in a week or two, especially considering they are creating so many components in various areas of technical environment, program design, program code, testing, and deployment. Use cases themselves need to be sliced.

Use cases can be sliced several ways, all desirable in various situations. A use case contains sets of paths, alternative and exception paths. The team can tackle the basic course of events first, then other paths later. This is a very common way to slice a use case. Alternative paths may be done separately, or grouped together when it makes sense. Exception paths are often grouped together, and are usually done after the basic course of events and most alternative paths are complete.

Another way to slice is to use "roughed-in" components. Let's say the Select Agent use case is undertaken early in the lifecycle, before any real work on the user interface design, including metaphor, widgets, standards, and color themes is ready. The Select Agent team may decide to "rough in" their user interface very simply, knowing that they will have to replace it later with the real thing. But at least they can get an end-to-end Select Agent use case working, with a visual (if not particularly appealing) user interface for the businesspeople to play with. Other components can be roughed in too: dummy database tables, software objects (interfaces defined but only roughed in for method implementation), package calls ( dummy interfaces), and so on.

The team will create the user interface, the business rules layer, data mapping, and database tables to make the use case work.

7.6.6 Tie Everything Back to the Business

Project managers who must manage every task and every assignment may not be able to tie those tasks specifically back to the original business needs. However, if project managers become focused on management by use cases, they can have a clearer traceability of the tasks back to the business requirements because the use cases are the business requirements.

Further, once a team has completed an iteration making a use case "real," the businesspeople can inspect the use case and make sure it accurately reflects what they need. In fact, "what they need" may even have changed slightly (or drastically!) between the time they provided those specifications and now. This gives them a chance to fold in any new requirements, which can be addressed in the upcoming iterations.

A businessperson will be much more comfortable seeing a project schedule that maps out when each piece of functionality (use case, non-functional) will be complete, rather than a task-oriented schedule (divided by technology or subsystem). Plus, he gets the chance to provide input regularly by playing with the real system, as it is built.

The value of this short feedback loop cannot be overstated. It is a tremendous way to stay on target, even when that target is moving. (Do they ever stay still?) Short feedback loops are the only way to stay on track in today's crazy world of business and technology.

Why not use a prototype ? There are several reasons. First, businesspeople need to interact with the real system. If they comment on some specific need for change and the team responds, "well, the real system will act differently," then the feedback path is lost. Second, remember the prototyping efforts you've been involved in. How many people were dedicated to maintaining the prototype? How much fun was it making sure that the prototype reflected the real system, and the real system was incorporating the changes in the prototype that were requested by the businesspeople? Not much. Allow the businesspeople access to the real system, and let the real system evolve under their fingertips.

7.6.7 Create Demonstrable Deliverables

This lifecycle of creating a feedback loop between the businesspeople and the real system is called creating "demonstrable deliverables." Businesspeople do not care about design documents. They do not care about paper or pictorial specifications. They care only about the end result. Does that mean you should forget about analysis and design? Certainly not. It means that when you want to show the businesspeople something, show them the working application. The running, tested code. Nothing less. Not prototypes , not screen snapshots. The only exception to this rule is that user interface storyboards can make sense early in the lifecycle before the first iterations have begun. But those storyboards should be created only after the use cases are far enough along that the businesspeople will not get confused by the user interface details introduced on the storyboards.

7.6.8 Learn the Art of "Good Enough" Quality

Microsoft knows how to build its products to a "good enough" quality level. The rest of us could learn from this.

There is something about Microsoft that is larger than life. Bill Gates is the richest person in the world. Microsoft is the largest and most successful software company in the world. The Windows operating system is installed on over 90% of all personal computers in the world.

Yet the quality of their software is quite poor. Yes, it is complex, but is that really an excuse for the number of crashes we all regularly experience using Microsoft products? There is a joke that if Microsoft built cars, we would all be crashing all the time, even if there were no other cars around.

Microsoft learned the rule of "good enough." While other software companies actually created better quality software, Microsoft learned that success depended on a combination of good enough quality and fast enough time-to-market .

This is a very painful lesson for software engineers . We have always been taught that creating error-free software is the ultimate goal. The closer we can get to that goal, the better. Deadlines, cantankerous users, moving requirements targets ”these are the enemy of error-free software.

As sacrilegious as it sounds, we have to shake this mind-set, especially in the HI/I lifecycle. The business does not want error-free software, no matter what they tell you. They are not willing to pay for it. They understand that, as we approach error-free software, the cost increases until, to get those last few bugs out, we approach infinite cost.

7.6.9 The Pieces Will Be Smaller Than You Think

Project managers used to dealing with large technology components or business subsystems as planning elements will be uncomfortable with use cases being much finer grained.

This fine-grained nature is actually a plus for planning. Project managers must resist the urge to "bundle" use cases together in order to assign them to teams.

7.6.10 Expect Negotiation, Not Specification

If asked, a businessperson will say that the number one skill in business is negotiation. Business cannot function without negotiation, and the better you are, the better you'll do in business. However, in the realm of business analysts, project managers, and technology specialists, negotiation is underemphasized. As a result, every interaction between businesspeople and software project team members is a mismatch! Excellent negotiators against people who don't even think it's part of their jobs!

Our advice here has two parts . First ”take a negotiation course! You'll need it if you plan to be an effective business analyst, project manager, or any other role that interacts with businesspeople. Second ”expect negotiation, not specification, from businesspeople. They come to each meeting expecting to negotiate. If the businesspeople state that some feature " absolutely must be in Release One!" that is most likely a negotiating ploy. Learn how to negotiate. In a HI/I lifecycle, you have the ability to trade requirements in and out of scope later in the lifecycle than with waterfall. Take advantage of that flexibility. The thing we hate to see is a dejected business analyst or project manager coming back from a meeting with the businesspeople saying,"Well, guess what they just added to the scope?" If they add something, ask them what they are going to remove. As we mentioned about HI/I, negotiate requirements out as your first tactic, and only when that fails, should you negotiate end dates and team size and development costs. It is much better to swap out requirements than to push out the delivery dates and demotivate the team.

7.6.11 Forget about Baselines and Sign-offs

Truces are needed between enemies, not partners .

”K ULAK AND G UINEY

Project baselines, like those produced in tools like Microsoft Project or Primavera, are not useful in a HI/I project. Since requirements are trading in and out of scope well into the project time line (perhaps as late as 70% of the way through!) the baselines built on certain tasks being in certain spots in the lifecycle become outdated quickly. Although a project management tool can be useful, baselining is not.

Sign-offs are a relic of waterfall projects. A HI/I lifecycle creates a tight feedback loop between the businesspeople and the software project team. It pulls their interest and participation fully into the project. The system becomes as much their system as the project team's system. Take advantage of that fact and remove the sign-off process completely. Sign-offs will only drive stakes into the ground to highlight the differences between the two sides.

7.6.12 Estimate by Doing

In a HI/I lifecycle, the estimates are refined after each iteration. The estimate at the beginning of the first iteration tackles iterations 1, 2, and maybe 3, but no further (depending on iteration length). Once the first iteration is complete, the estimates for 2 through 4 will be clearer. Iteration 2 will be crystal clear, 3 a little less clear, and 4 just about in focus. Continue this, learn as you progress, and re-estimate after every iteration.

How does this work for fixed-price, outsourced projects? It doesn't, quite frankly! We will define a typical fixed-price project as a project that has fixed cost, schedule, and requirements from the very beginning. An outsourced project is one where the software lifecycle is performed primarily by a third-party software consulting company.

If you have ever been on a well-run fixed price, outsourced project, call us. We haven't had that experience. These projects are extremely problematic , antagonistic between client and contractor, and almost always a disaster for all parties involved. We are not sorry to drive another stake into the heart of the fixed-price project. Software projects are complex enough that they must be partnerships between a client and a contractor, with an agreement that requirements will change throughout, even though the budget and schedule might be limited for business reasons. To fix all three is insanity. But you knew that, didn't you?

As far as outsourced software development, our opinion is that third-party consultants must be used to coach, train, and mentor internal IT department employees , not to do the work themselves. So many times we've seen consultants sweep in to develop a new application and then leave, with the internal staff unaware of how the application works and how to maintain it. The one exception to this rule would be if the client company is too small to have its own IT department, in which case they should likely be looking at packaged solutions instead of custom software.

7.6.13 Calculate Return-on-Investment in a New Way Using Portfolios

Return-on-investment (ROI) is an important factor for another software project. How do you calculate it? There are several factors to evaluate:

• Time saved by the manual processes automated

• Time saved by faster processing using software

• Competitive advantage gained by managing complexity better through software

However, there are lots of intangibles that are hard to measure. If the new application replaces an old one, how can you measure something like better usability? Measuring every mouse movement and keystroke is possible but unlikely to be accurate.

Other applications offer indirect ROI. An application allowing a network security administrator to protect against hacker attacks has ROI, but how would you measure it? The projected cost of a hacker attack, the likelihood of the attack, versus the cost of the software? This would be a difficult task at best.

A new method of ROI calculation has arrived called portfolio management . It allows ROI to be viewed as the extent to which risk was mitigated for the enterprise. For extensive information on portfolio management, see The Real Options Solution ”Finding Total Value in a High-Risk World (Boer 2002) and the article at http://www.portfoliomgt.org/read.asp?ItemID=1222.