Practice 20. Continuously Reevaluate What You Do

Heath Newburn

If we always challenge what we do and seek innovative ways to develop software, we can improve how we work and achieve better project results.

Problem

In life it's often the actions we don't take rather than the deeds we do that cause us the greatest problems. A real danger any software project faces is the inertia and mediocrity of the mundane. We're usually so pressed for time in our rush to get our applications out the door while checking all of our process boxes that we don't have time to look for ways to improve how we work. Unless there's an edict from the top or a fire to put out, we rarely look at how we can or should improve our methods. This inertia leads to more of the same, which can be costly to our projects.

Background

We often make unjustified assumptions. Take this story, for example. A grandfather sat down to lunch at his grandson's house to celebrate the grandson's promotion to vice president at his company. The younger man knew that porterhouse steak was his grandfather's favorite, so he brought his grandfather a four-ounce steak just like those he had always seen his father make. The older man looked up in surprise and grinned: "Cheap, just like your dad." The grandson turned a little red and asked, "What do you mean, Grandpa? I thought you liked your steaks small; that's how Dad always makes them, and you taught him." The grandfather laughed and shook his head: "It's not that we liked small steaks, it was just all we could afford!"

Sometimes we make assumptions about the "best" or "right" way to do something because that's the way we've usually seen it done. Often, in fact, it's the only way we've seen it done. But as the example above illustrates, that doesn't mean our assumptions are right.

When I moved back to Austin after a four-year absence, I started taking what I knew to be the shortest route to my new workplace. A few months later, I had to run an errand on the way home and encountered a new road that hadn't existed when I left town, though I later found out that it had been there for more than a year! That accidental discovery saved me 20 minutes a day, and cutting more than an hour a week off my commuting time made a real difference to my life. The way I had thought was best was no longer best, since my assumptions were now wrong!

Within our development process we often do the same thing. The way we've done things in the past has obviously worked to some degree, so what's the need for change? If it ain't broke, don't fix it, right? And while development processes often become a religious discussionwhether it's spiral, RUP, XP, or some other form of agile developmenteverybody's got a reason to use or not use a particular approach.

What's important for you to understand is the best way to leverage your process. What changes do you need to make to your process to make your project successful? For the activities and artifacts that you can influence the most, what should you look at changing now, and how should you drive those changes?

Applying the Practice

Continuously refining your process is critical to the long-term success of your team and project. You can start looking at all the parts of your process by asking "Why do we do it that way?" Once you understand why, you can quickly eliminate problems by verifying that the assumptions that have been made are still valid.

Start by Asking Questions

One of the most effective ways to begin reexamining what you're doing is to start asking questions. "Why" questions can be particularly effective. If you've ever been around a small child, you know that at some point the constant questions can become infuriating, but they often turn out to be gems worth pondering. "Why do you work so late, Daddy?", "Why do I have to wear clothes?" (my son's current favorite), "Why do we sleep?", "Why do I have to go to bed early?" . . .

Such questions help us evaluate our habits and actions. Why do we do the things we do? Are they the best choices for us? We may take a route to school that saves us five minutes, but if the more leisurely route boasts better scenery, has no stop-and-go traffic, and is free of aggressive drivers, are we making the right decision?

Once we understand the "why," we can address the "how" and the "what." If our goal is to get to school in the shortest amount of time, then we are indeed making the right choice. If our goal is to use the trip to talk to the kids in the car and enjoy a relaxing start to the day, maybe we should take a different approach. Almost all of us take the same route to work every day. Is it always the best way, even on those days when there's an accident? Are some routes slower than others depending on what time of day it is? Or do we just end up taking the same route every day because that's what we did the day before? How many things do we do because that's the way we did them the day before?

Once we understand the "why," we can address the "how" and the "what."

Continuously Verify Assumptions

Throughout the project, we should continuously challenge what we do by verifying assumptions. As we progress through a project, we have to ensure that our assumptions, that is, what we believe to be valid and true, still are. Or, as Allen Carter^[7] asked in thousands of English classes at Clayton High School:

^[7] High school humanities teacher in Clayton, New Mexico.

• What are we doing?

• Why are we doing it?

• What is it good for?

• How do we know?

These questions could constitute the core of a checkpoint for every iteration in your process.

Verifying the integrity of your project at regular checkpoints can be extremely useful. If nothing else, it's worth asking whether or not the project should be continued. A classic example is the Iridium satellite project. This $6 billion project was undertaken for one simple reason: to allow you to carry a single phone with which to make or receive a phone call anywhere in the world. In the early 1990s, phone coverage was a concern for global travelers. Executives, salespeople, oilfield workers, and others had to either rent phones at outrageous rates, own multiple phones, or (horror of horrors!) be out of touch. With a huge number of competing cell standards, large areas with no cell coverage, and no clear leader who could drive adoption of a worldwide standard, the idea of a satellite system seemed like a winner. For Iridium, launching more than eighty satellites to cover the world would be an expensive proposition but would fill a gap for all of these potential customers with big pocket books. However, while the company was setting up the system, the world quickly changed. As GSM became the European standard and no-cost roaming covered the United States, other areas of the globe fell into line; pretty soon the reason for Iridium's existence became moot. By focusing only on the end goal, the project headed for rapid doom, and the system was eventually sold for pennies on the dollar. If someone had challenged, "What are we trying to accomplish here?" as the business environment was evolving, they would have changed the business plan before spending billions of dollars.

Always Be Challenging

A good way to challenge your processes, your assumptions, and your habits is to build a paradigm for asking the right questions. Simply asking why isn't enough. If nothing else, you'll end up seriously annoying the people around you and making yourself nuts. As you look at your process and work flows, if you hear or see something that doesn't seem quite right, use a questioning approach to engage discussions and ensure that you're on your current path for a real reason, rather than just for lack of a reason to do something else. Whatever choices you've made, you must be able to articulate a reason for them.

While Carter's four questions above are great, you need to be able to answer not only if it's the right thing but also if it's the best thing. Edward de Bono describes several approaches to evaluating any situation in his ABC (Always Be Challenging)^[8] approach (see Figure 7.4). Whenever we look at a particular situation, we have to determine three things. Being a lateral thinker, he starts backwards.

^[8] de Bono 1970.

• CCut: Is there still a need to do what we're doing? Is the goal still relevant? Do we now have other priorities? Are our assumptions still valid?

  Cut what is not worth doing, validate the why, and improve what you need to do.

  
  

• BBecause: What are the reasons that we've adopted this approach? Are we doing the same thing because that's the way we've always done it?

• AAlternatives: if there is no other approach, how can we refine and improve the one we have to make it more effective or efficient?

Figure 7.4. The ABCs of Improvement: Always Be Challenging.

To improve the way you are working, you should always challenge current practices. Using Edward de Bono's ABCs, cut out what is not worth doing, validate the reason you're using your current methods, and improve the way you are doing what you need to do if there are no alternatives.

This approach works well when we look at our test methodology at the beginning of a project. We'll look at what we've done in past projects and determine whether it will work for this project. If the reasons that we took the particular approach in the past are still validthat is, we have to do testingand we have the same skills, same personnel, and same sets of problems, we might still take this approach. However, there may be new tools available that will allow us to enhance our capabilities. So we'll add some additional training cycles and make sure we can show our reasoning behind the decision to move forward on this new path, rather than just blindly doing the same things we did last time.

"Sunset" Processes

Another way to force rethinking is by "sunsetting" processes. Governments often "sunset" laws or committees, which means that those laws or entities expire after some period of time and can be renewed only by an overt act. This strategy forces the owners of sunsetted items to continue to justify their existence and provide evidence that they are actually adding value. If you can't articulate the value, then you can probably safely eliminate the item. Finding the right way to establish the value proposition in this method, or any other described here, can be a big challenge. Knowing how to measure success is the key to knowing what to keep, what to change, and what to lose.

If you can't articulate the value, eliminate the item.

Measure Success

The issue of measuring success brings up a crucial point. It's important to understand just how to determine what parts of your process are working. This is a conundrum for process owners. Although you don't want to drown people in metrics and measurement, you have to have some method to determine the effectiveness of the process. While the method can be subjective, it's much better to determine and focus on what can be objectively measured. Examine the work products from the process; how can they be quantified and qualified? It is also important to set up measures to drive improvements of your weakest area(s) within the overall system, rather than improvement of an area that is already strong.^[9]

^[9] Goldratt 2004.

Figure 7.5. Measure Success.

To know which process to keep and which to throw away, you need to know what characterizes success. Measuring success crystallizes which processes are worth keeping and which are not. Is it better with many small fruits, or a few large fruits?

One common measurement of success during a project is the quality of code, as defined by the number of defects discovered. This measurement is often taken by comparing a defect projection rate against the actual number of defects. If the number of defects in your project exceeds the target by 300 percent, is the project successful? Before you say no, what if you had a test team whose skill far exceeded that of previous testers? What if the defect projections themselves were flawed?

In other words, it's not enough to have a single measure of success. You might also rate code quality by the number of blocked test cases, customer satisfaction surveys from early release programs, depth and breadth of execution of automated test buckets, or many other factors. Most measurements will be defined in terms of the constraints of quality, cost, and speed, all three of which need to be measurable at any time. This may mean cost and budget analysis, alignment with milestone delivery dates, the amount of overtime being burned on a project, or usability test scores. While any number of methods may be relevant to your project, make sure you're not evaluating in a single dimension. Overlapping measurements will ensure that you have a better understanding of your project.

Whatever metrics you choose to measure your product, make sure that those measurements are readily accessible and easily gathered by your teams. One of the things that can become a huge drag on team leads is metrics reporting to different people. I've been on teams where we had to produce six different sets of charts to satisfy the measurement required by all the layers of management. One team lead finally got fed up, created a two-page chart that could be generated by tools, and advocated it all the way to the executive level until he got buy-in across the board. By focusing on what was really needed in his status reporting and being able to gather it automatically, he greatly improved the quality of his work life and that of his team. With these measurements in hand, he could easily show what was and wasn't working and enabled the leadership team to make the right decisions.

Make sure that measurements are readily accessible and easily gathered.

Do Iteration Reviews and Retrospectives

One of the benefits of iterative development is that you have an opportunity to assess the effectiveness of your team and the process you use at the end of each iteration. You can then use what you have learned in the review to improve the way you work in the next iteration.

In iterative development you assess your team and process at the end of each iteration.

The term retrospective^[10] is frequently used to describe a review focused on learning how to work more effectively. As an example, the Agile Manifesto states^[11] "At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly." We have found that a retrospective is best done at the end of each iteration together with an assessment of the results of the iteration. This review often creates a sobering awareness of potential problems the project may face. For example, if you realize that you implemented the capabilities you had planned to implement within the iteration, but the quality of the code is so low that it is close to unusable, you gain a lot of insights into where you can look for improvements in the way you are working. You can probably understand what improvements are required by analyzing your test approach, the amount of time spent on using the application within each iteration, and the prioritization you put on developing new capabilities versus improving the quality of the application.

^[10] Kerth 2001.

^[11] www.agilemanifesto.org/.

When conducting a retrospective, make sure you focus on how to improve, not on blaming people for not doing a good job, as discussed in the section Build Trust in Practice 12: Build High-Performance Teams. To address this issue, Norman Kerth defines what he calls the Prime Directive for retrospectives^[12]: "Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand." He also defines four key questions to focus on during the retrospective:

^[12] See Kerth 2001.

• What did we do well, that if we don't discuss we might forget?

• What did we learn?

• What should we do differently next time?

• What still puzzles us?

To ensure that the retrospective is a good learning experience, Ellen Gottesdiener^[13] defines three required parameters:

^[13] Gottesdiener 2003.

• Immediacy. Make sure that you can apply lessons learned within days of the retrospective. This fits well with doing the retrospective in conjunction with the iteration assessment.

• Relevance. Make sure that the learning deals with topics that team members care about, such as how to improve test efforts, collaboration across various functions within the teams, or how to be more effective in the build process.

• Self-direction. Make sure that all team members take ownership and control of their learning and actions to make improvements. Make sure that a plan is produced during the retrospective outlining who should change what.

Note that it is important to conduct retrospectives at the end of each iteration rather than, as many projects unfortunately do, only at the end of the project. Waiting until the end of the project creates several problems:

• It is at this point too late to improve upon your current project.

• Teams often disperse at the end of the project, and lessons learned from one project may not apply to the next project, where you may have very different team dynamics and issues.

• If teams disperse at the end of the project, team members are less likely to put the required energy into the brainstorming session, since they may not benefit from the lessons learned.

• Technology issues addressed in one project may not be relevant for the next project, which may use other technology.

• There is no incentive to apply lessons learned when the project is over, and in most cases, nobody sees it as his or her responsibility to make effective use of the lessons learned.

Figure 7.6. Iteration Assessment.

By assessing the processes and results achieved at the end of each iteration, team members can understand how to improve the way they work for the remainder of the project.

Don't Go to Abilene Just Because Everyone Else Is Going

There are lots of methods that assist in decision making: risk avoidance mechanisms, Monte Carlo analysis, idea generation techniques, causal analysis, and many others. Find a method that is repeatable and useful for your team and use it. Again, making a decision based solely on lack of failure with a past experience isn't optimizing your talent pool. By the same consideration, "going along to get along" is a road to hell paved with good intentions.

"Going along to get along" is a road to hell paved with good intentions.

A death spiral for any group is to get caught up in group-think or appeasement. You have to challenge assumptions when they don't look very smart. Professor Jerry Harvey described what he called the "Abilene paradox," based on a real-life event that might sound familiar to some of us.^[14]

^[14] Harvey 1996.

Figure 7.7. Challenge Assumptions to Avoid the Abilene Paradox.

In many everyday situations we do things a certain way just because we assume that everybody thinks this approach is a good one. As we can learn from the Abilene paradox, we may find that nobody thinks what we are doing is a good idea, but that nobody ever raised the issue of whether the process should be followed.

A woman takes her East-Coast-raised husband home to West Texas in the middle of a blistering summer. The family spends most of the days on a nice screened-in porch with ceiling fans and ice-cold lemonade to keep the heat away, and much to the son-in-law's surprise the atmosphere is actually very pleasant and conversational. One day the father-in-law, noticing a lull in the conversation, asks, "Y'all want to go to Abilene for dinner?" No one seems in disagreement, so they all pack into the car and head down the dusty road.

After a three-hour round trip with no air conditioning in a broiling sun, choking dust, carnivorous horse flies, and a greasy meal that some prisons wouldn't serve, they all clamber back onto the porch at the homestead. They're grumpy, sweaty, and generally miserable. The daughter notes that the meal wasn't much and maybe Mom could get out some of her fried chicken from lunch. The father and son-in-law begin to argue over whose meal was worse. After a little bit of lemonade, they cool down some, and the father-in-law remarks "Yep, I can't even believe y'all wanted to go." The others look on in shock. No one remembers pressing the idea, and it was clearly the father who proposed it. The father shakes his head defensively. "I thought you were bored, and I was just trying to be polite! If you didn't want to go, why didn't you say so?" Everyone descends into silence and private fuming. Because everyone had wanted to be polite, no one had voiced any disagreement, each thinking that only he or she didn't want to go.

The moral here is that in your product directions, development process, and day-to-day decisions, make sure you're not headed to Abilene. Ensure that you're going somewhere for a reason, rather than just going along for the ride because that's what you think everyone else wants.

Closing Thoughts

Allowing your decision-making process to be governed by the laziness of "that's the way it's always been done" is a bad road to take. Create positive change rather than just being carried by the inertia of your organization. Ensure that you've examined the rationale behind your actions and that you can clearly articulate what you are doing, why you are doing it, and how you know it's valuable. Avoid the pitfalls of the Abilene paradox and don't blindly do what you did before. By making sure that you're actually adding value to your project and not just following the well-worn path, you'll avoid the "good enough" syndrome and make your project more successful.

Other Methods

If you follow a waterfall process, you will at least theoretically do requirements, design, implementation, integration, and testing only once and in a sequential order. This process therefore provides you with limited ability to learn from your mistakes, because if you determine after the requirements phase that you can improve the way you work with requirements, you cannot apply that knowledge until the next project.

Unlike waterfall development, iterative development gives you the excellent ability to improve what you are doing in each iteration through iteration review and retrospectives.

Agile development approaches, such as XP and Scrum, clearly emphasize the value of continuously improving. Both of these methods recommend retrospectives at the end of each iteration. Scrum calls these Sprint Reviews. Scrum also prescribes daily Scrum meetings, where three questions are answered by each team member:

• What have you done since last meeting?

• What will you do between now and next meeting?

• What obstacles stood in the way of doing work?

The last questions provide the Scrum Master with information about what stands in the way of progress, so that obstacles can be removed and related improvements to the working procedure can be implemented as appropriate.

XP also prescribes pair programming, where two developers work side by side to develop parts of the application, enabling both to learn from each other.

RUP and OpenUP/Basic articulate the importance of doing reviews at the end of each iteration, milestone reviews at the end of each phase, and a review at the end of the project to learn how you can work more effectively. RUP and OpenUP/Basic include daily status meetings and pair programming as optional techniques you may consider using. RUP and OpenUP/Basic also have a built-in review step for most tasks and checklists for most artifacts, allowing the team to continuously improve both the quality of artifacts and how to do things. It should be noted that the value of reviews diminishes as collaboration increases, and that collaboration may therefore have a better pay-off than more reviews.

Levels of Adoption

This practice can be adopted at three different levels:

• Basic. Work product owners reevaluate parts of the process on an ad hoc basis. This should lead to more effective processes with less overhead, and hence also the opportunity for shorter iterations.

• Intermediate. The team carries out end-of-iteration reviews and retrospectives and uses the results to continuously improve the process. This should lead to more effective processes with less overhead, and hence also the opportunity for shorter iterations.

• Advanced. A governing body ensures that the entire process has had active reviews to verify that each part of the process is the most useful for the team. All team members continually strive to find better ways of building their part of the solution.

  The governance introduces more overhead, and hence more discipline and longer iterations, but this system should in the end lead to a process that better addresses the specific needs of the organization.

Related Practices

• Practice 2: Execute Your Project in Iterations shows that using iterative development allows you to continuously reexamine what you're creating.

• Practice 12: Build High-Performance Teams discusses building trust, which is crucial to enabling all team members to discuss openly and constructively how to improve the way they work.

• Practice 19: Rightsize Your Process explains how to work more effectively by focusing only on the activities in your process for which the benefit is greater than the cost. Continuously improving your process is core to reevaluating what you do.

Additional Information

Information in the Unified Process

OpenUP/Basic provides guidance on iteration reviews, retrospectives, pair programming, and other basic and intermediate practices. RUP adds guidelines on how team members can assess the artifacts they produce and provides guidance on process improvement.

Additional Reading

Books on improving software development processes:

Alistair Cockburn. Agile Software Development. Addison-Wesley, 2002.

Norman L. Kerth. Project Retrospectives: A Handbook for Team Reviews. Dorset House, 2001.

Relevant books outside the field of software development:

Edward de Bono. Serious Creativity: Using the Power of Lateral Thinking to Create New Ideas. HarperBusiness, 1993.

Edward de Bono. Lateral Thinking: Creativity Step by Step. Perennial, 1970.

Jerry Harvey. The Abilene Paradox and Other Meditations on Management. Jossey-Bass, 1996.

Eliyahu M. Goldratt and Jeff Cox. The Goal, Third Edition. North River Press, 2004.