Practice 3. Embrace and Manage Change

Per Kroll

Embracing change helps you to build a system that addresses stakeholder needs, and managing change allows you to reduce costs and improve predictability.

Problem

Physical systems, like bridges and skyscrapers, are very difficult to change. Adding a lane to a bridge or expanding a building is extremely costly once construction has begun. For this reason, detailed planning is critical to the success of physical engineering projects.

Software, however, can be changed far more easily. Software can be designed to be configured. Early prototypes can demonstrate behavior, which can then be modified for the final system. Rather than try to design software in detail up front, it is often far better to build some software and use the working product as a communication medium to help determine what is really needed.

Use the working product as a communication medium to help determine what is really needed.

Nevertheless, although the ability of software to support changes provides a great opportunity to evolve it to meet changing stakeholder requirements, continuous change presents a challenge to software development teams. Changes can introduce costs that need to be managed. They can also cause developers to thrash around, unable to progress with the software because the requirements are changing faster than they can keep up.

This practice describes how to manage change and keep chaos at bay while embracing necessary and important changes.

Background

My wife and I are currently in the middle of a garden and house renovation project. We started by writing down a rough sketch of what we wanted to have donea new deck, a large balconyand some rough ideas for landscaping our garden. We discussed the project with a few different contractors, asking each to come up with a design. We picked one design (with an associated price tag), and worked with that contractor to refine it to our exact requirements. The contractor was very open to making whatever changes we wanted to ensure that we would be happy with the end result, and we threw around a lot of ideas, while sticking to the overall agreed-upon price.

As the project progressed, we ran into unexpected problems. We did not get a permit for building as large a balcony as we wanted, so we needed to change the plans and design a smaller one. Then, once we changed the structure of the balcony, we decided to alter slightly the dimensions and placement of our deck. At a later stage, our neighbors insisted we plant a few trees as a condition for their approval of the construction of our balcony. All these changes were reasonable, as they did not alter the fundamental parameters of what we were trying to achieve, but some resulted in changes that needed to be approved by all stakeholders, that is the contractor, my wife, and me. For these the contractor produced a "change order," describing in one or two sentences what revisions were being made to the original plan, which we all signed. Sometimes the change order meant a revision to the overall cost of the project, sometimes it did not; but it ensured that we all agreed on what needed to be changed.

As the construction and landscaping project progressed, other, more minor issues needed to be resolved, such as what flowers to plant, exactly where to put plants, what type of stain to use on the deck, and so on. Most of these decisions could be handled without a change order, but for all major revisions we used a change order, not only to make sure that we all agreed on the changes but also to allow us to go back later and understand the end result.

Sometimes my wife and I wanted changes that would have been very expensive. When we first saw the size of the deck, after it was almost completed, we felt that we had gone overboard with the size. Maybe we should have more grass, and less deck. A quick discussion with the contractor made us realize the cost of introducing such a change so late in the project. We decided that we should move ahead with the plans as they were. Project members need to have the guts to inform a customer honestly of the impact of a change (see Figure 2-4).

Figure 2.4. Some Changes Can Be Very Costly.

The Swedish navy launched a new grand royal flagship, Vasa, in 1628. The Swedish king asked the builder to add a second gun deck after the project was initiated. Nobody dared to tell the king about the associated risk, even though those involved knew that this change would make the ship too top-heavy to be seaworthy. The Vasa sank on its maiden voyage. Always remember to provide your customer with reliable information about the potential impact of a change.

Major changes can be made early in a project at a very limited cost.

Running a project like the one above is similar in many ways to running a software project. So what can we learn from the above?

• Major changes can be made early in a project at a very limited cost. Major alterations carried out later in the project become increasingly expensive to implement. Note that some people in the agile community claim that changes can be made at any time with no cost increase. We will give several examples in this practice showing why this assertion is untrue. Agile techniques, including the ones introduced in this book, aim at reducing the cost of making changes, but they will not completely eliminate the cost of most changes, nor will they allow you to make any type of change at any given stage in a project without potentially serious negative effects.

• To satisfy customer needs, you typically need to introduce a fair amount of change to the original plans. You also need to inform the customer fully of the cost and schedule implications of carrying out changes (especially late in the project) and allow the customer to use this information to determine whether the change is really necessary or not.

• For certain projects, such as those with a contract, as in the construction and landscaping example above, you may need to formalize the changes; in other cases, informal discussion with key stake-holders may be enough. In general, less formality is needed regarding changes at the beginning of a project, but as the project progresses, you should document requested changes with appropriate formality to ensure that their impact is properly understood and agreed on by all stakeholders.

Applying the Practice

In this practice we will walk through concrete guidelines allowing you to embrace and manage change more effectively. We will first look at different types of change and their associated cost profile and then examine how the Unified Process lifecycle can help you to manage change. We will describe how to adapt the process of managing change to the characteristics of your project and the phase you are in. We will then review some techniques to maximize change freedom, provide guidelines for reducing thrashing by minimizing change within an iteration, and explain how to increase productivity by fixing defects as they are found, rather than maintaining a backlog of defects to be fixed sometime in the future. Let's take a closer look at these topics!

Understand the Cost Profile of Different Types of Changes

Understand cost profiles to avoid costly changes at a late stage.

Different types of changes have a different cost based on when in the project they occur, and it is important to understand the cost profiles when prioritizing project work to avoid costly changes at a late stage. Let's look at some examples.

• Major changes to the business solution. Major change to the business solution involves extensive rework of requirements to address a different set of users or user needs. As an example, let's assume that you plan to replace three existing applications with one new application. The cost of replacing each application is $25,000. If on day 1 of the project you decide not to replace one of the applications, the cost is close to $0. If instead you make that decision on the last day of the project, the cost is $25,000 (assuming you cannot reuse the code later on). There is thus a fair amount of flexibility in making this type of modification early in the project, before you have started to invest in requirements, design, implementation, and testing of a specific business solution.

• Major changes to the architecture. Major architectural rework is frequently very expensive when done late in the project. Examples of such changes include switching from a rich-client to a thin-client, making a buy-versus-build decision for a major component, or changing strategy for recovery of a safety-critical system. If you decide to make such a change late in the project, you may have to rework vast amounts of code, potentially delaying the project. It is therefore important to validate architectural decisions by partially implementing the application early on. By realizing early what architectural decisions may be suboptimal, you can modify them early on and avoid later rework. Other modifications to the architecture can be made inexpensively. For example, by isolating persistency to be handled by a persistency layer in your architecture, you can change how to deal with persistency with minimal impact to your application at a reasonably late stage. One goal of architectures is to minimize the cost of future changes. Refactoring can also reduce the cost of change, but it only goes so far, as discussed in the section Techniques Maximizing Change Freedom on page 68.

• Change to the design and implementation. If you do component-based development, leverage service-oriented architectures, or use other well-structured design approaches that localize the impact of change, you can make limited changes to your design and implementation at fairly low cost up until the end game of the project, when you need to stabilize and fine-tune your application.

• Reduction of scope. The cost of cutting scopeand hence postponing features to the next releasecan be relatively inexpensive throughout the project. This method assumes that you use an iterative approach whereby you develop the most essential capabilities first, and that you cut scope by de-scoping the least essential capabilities first. Project teams should use de-scoping as a key tool to ensure on-time project delivery of the most essential capabilities.

Focusing early iterations on building the prototypes, developing and testing key capabilities, and achieving close collaboration with stakeholders will reduce the likelihood of costly shifts in direction late in the project.

Focusing early iterations on prototypes and collaboration with stakeholders will reduce the likelihood of costly shifts late in the project.

The Unified Process Lifecycle and Change

The phases in the Unified Process lifecycle are designed to force the costly changes discussed in the previous section to occur early (see Figure 2.5), while allowing you to make critical decisions as late as safely possible, as discussed in the section Techniques Maximizing Change Freedom.

Figure 2.5. Phases Are Optimized to Minimize Overall Cost of Change.

The cost of introducing a change varies according to the lifecycle phase, and each type of change has its own cost profile. The Unified Process lifecycle is optimized to minimize overall cost of change, while maximizing the freedom to make changes. In general, as the project progresses, you should be more careful about introducing change, especially to the overall business solution and the architecture.

• The Inception phase aims to drive out risks associated with major changes to the business solution. Its purpose is not to force detailed specification or prevent later changes to requirements.

• The Elaboration phase aims to drive out risks associated with major changes to the architecture by validating a viable architecture through implementation of a minimum amount of code. The purpose is not to prevent later changes to the architecture, but to minimize costly changes late in the project.

• The Construction phase aims to drive convergence around what constitutes an acceptable operational version of our application, with continuous feedback from early versions of the system leading to the release of a beta version of your application by the end of this phase.

• The Transition phase aims to drive out the last issues with the release, with the primary goal of stabilizing the release and addressing last-minute concerns from stakeholders. There may be incremental releases in Transition.

The above guidance should not be interpreted as preventing you from carrying out the necessary changes in a later phase than desired. For example, if you notice during Construction that you need to rethink the business solution and make extensive changes to the requirements to be able to deliver a satisfactory application, naturally you should consider implementing that change. But during Inception, you would not really think twice about it. If you need to change the business solution during Construction, it is reasonable to consider the following actions:

• Understand the cost of change.

• Understand the consequence of not making the change. If the impact is severe, it warrants taking on a big cost for the change.

• Formalize the decision around making the change to ensure that it is not done on a whim, or without appropriate management control. The following section elaborates on this step.

• Analyze whether you can improve your process to avoid these late and costly changes in future projects.

Adapt the Change Management Process Suited to the Phase and Project Type

For some applications, the cost of making any type of late change can be quite prohibitive.

For some applications, the cost of making any type of late change can be quite prohibitive. Consider the following:

• A safety- or business-critical system where you cannot accept defects. A late change could compromise the quality, and the resulting cost or impact may not be acceptable.

• A broadly deployed shrink-wrapped product. A shrink-wrapped product that is broadly deployed typically has a lot of supporting collateral, a large number of people who need to be trained in services and support organizations, extensive beta testing, and localization to many languages. All of these things make it expensive to introduce late changes.

• A technically highly complex application, or an application developed by a very large team. Due to its complexity and size, testing may be quite comprehensive, requiring a considerable amount of time to stabilize the application.

• An application with a large number of stakeholders that need to be consulted before major changes are made. When changes are carried out late in the project, there may not be sufficient time to allow every stakeholder to be heard.

In the section Understand the Cost Profile of Different Types of Changes we also saw that major changes late in the project, especially to the business solution and the architecture, can be particularly costly. The above are examples of project types for which you need to formalize changes late in the project. So how should you go about formalizing the decision to make a change? Change approvals are carried out through a combination of manual procedures (for example, through Change Control Boards), and through tools such as Configuration and Change Management software.

Early in the project, you should ensure that the right stakeholders are aware of the change. This may require nothing more formal than an e-mail going out to appropriate stakeholders, or frequent demonstrations of working software to the stakeholder that allow you to show the changes already made and mention future planned changes, providing stakeholders with the ability to interject if they so desire. For larger and more formal projects, you may need to set up separate meetings to ensure proper handling of issues requiring contract renegotiations, or to guarantee that the right procedure is being followed from an auditing perspective due to FDA^[20] or other regulations.

^[20] Food and Drug Administration in the United States of America.

As projects near the end, you often want to make sure that you properly assess the potentially negative impact of any changes. For small projects, for example, a quick daily meeting may suffice to discuss major changes briefly and enable each team member to express any concerns. Larger projects, or projects requiring more formality, may instead leverage a Change Control Board with representatives from all key constituencies. This board discusses potential changes to make sure that the board members understand the pros and cons of any change. The board will then agree on whether or not to carry out the change.

As projects near the end, you want to assess the potentially negative impact of any changes.

Through configuration and change management tools, you can set up rules governing what changes can be made and by whom, and what process these changes need to follow. As an example, you may specify in your change and configuration management tooling that all defect fixes in the Transition phase require approval by a certain board or person before they can be included in a build.

Techniques Maximizing Change Freedom

The underlying assumption with iterative development is that the only way to build the right application is by getting feedback and then making changes to the solution to accommodate this feedback. Even though changes late in the project can be costly, you want to maximize your ability to accommodate change. Here are some techniques proven to maximize change freedom:

• Partial implementation. You should always strive to implement capabilities only partially before you get feedback, so that you can improve upon the solution before investing too much time on a capability. Consequently, you should implement perhaps only 10 percent of the code for a use case before you show it to end users for feedback. You will then have a good vehicle for discussing what the requirements really are for the use case and will be able to make changes to the use case with a minimum of rework.

• Refactoring. The cost of architecture or design changes can be reduced by continually improving your architecture, design,^[21] and data^[22] through refactoring. The principle behind refactoring is that the first design should address only the requirements you know you need today. As requirements evolve, you will also evolve the design in small increments to address additional capabilities. You may therefore first develop a design to accommodate a single-user scenario of an application and then gradually improve the design to provide multi-user support. Refactoring will not, however, free you from the need to make the right difficult design decisions early on. For example, if you make the wrong build-versus-buy decision and then invest 200 hours building a component that you later notice could have been completed in 20 hours by leveraging an open source component, refactoring will not allow you to regain your 200 hours. It will, however, allow you to refactor your design incrementally from leveraging your home-grown component to leveraging the open source component.

  ^[21] Fowler 1999.

  ^[22] Ambler 2006.

  If you cannot clearly articulate how not having a piece of information will hurt you, remove it.

  
  

• Keep documentation, design, and code simple.^[23] The more documentation you have, the more costly it is to maintain. You should therefore strive to keep your documentation to a minimum. However, having too little documentation can also be costly by leading to miscommunication or endless discussions revisiting issues already solved, so aim to strike the right balance based on your project needs. As a rule, if you cannot clearly articulate how not having a piece of information will hurt you, remove it. The same reasoning applies to your design. Choose the simplest design that addresses your needs. Continually strive to keep your design and code simple by refactoring it and cleaning it up, making it easier to make future changes.

  ^[23] Kent Beck describes this technique in Beck 2004.

• Predict future likely changes. By spending initial time on assessing likely changes, you can architect your application to minimize the costs of those changes, should they occur. For example, if you are unsure of what persistent storage mechanism you will need, first implement the simplest possible solution, such as file-based storage, and architect your application so that you can easily upgrade to a high-end database solution. You can document such potential changes in your software architecture document, or by using change cases.^[24]

  ^[24] See Bennett 1997.

• Effectively manage change requests. During a project, a lot of people inside and outside the immediate project team will have suggestions for how to improve your application. To leverage this feedback effectively, you need to make it easy for people to suggest an improvement. One solution is to make it easy for people to enter change requests into your change request database. If you are afraid that many suggestions may not make sense, you may choose to apply some type of filter, such as having a customer representative continually interact with key stakeholders, and they enter change requests based on feedback.

• Automate change management and change propagation. The cost of change can also be reduced by having procedures and tools in place for managing change. Automatic synchronization between design and code is an example of reducing the cost of change through automation, since it allows you to make some code changes working on a higher level of abstractionthe design. Another example is to have tools that allow you to couple a change request to all the right versions of files impacted by it. Once the designer has implemented a change, a tester can more easily verify that it has been properly implemented by gaining direct access to affected files.

• Organize your work to understand the impact of change. A change to, for example, a requirement, may also impact the design, code, user manuals, online documentation, training material, test assets, and so on. You should organize your work to make it easy to understand what impact a change to one artifact may have on other artifacts. You can to some extent achieve this goal through traceability in tools, as well as by following a use-case-driven process. If you have a change to a specific use case, you will be able to see directly its impact on your design (through corresponding use-case realization), on test cases (by organizing test cases by use case), on manuals (by having one user manual chapter for each use case), and so on.

Minimize Changes to the Current Iteration

Introducing changes within an iteration may lead to churn and thrashing.

As you work in an iteration, you will get a lot of good feedback on capabilities that you should change, and you will realize how you can do things better. You need to be very careful about introducing changes within an iteration, which may lead to churn and thrashing, thus preventing successful completion of the iteration. Here are some guidelines to consider:

• Remember that the next iteration is only a few weeks away. Any major changes should be listed and prioritized with all other items considered for the next iteration. If it is an important change, it will go into the next iteration, so it will soon be taken care of. If it is not an important change, you should be really glad you decided not to work on it within the current iteration. Examples of changes that fit within this category are adding a new feature or rethinking the main flow of a use case. These changes would typically impact the iteration plan you established at the beginning of the iteration, listing what you want to accomplish in this iteration.

• If it will take less time to implement a minor change directly than to document and discuss when to implement it later, just do it. Examples of such changes include minor changes to use cases or screen layouts. These changes typically do not impact the iteration plan.

• When in doubt about whether something is a minor or major change, make a note of the change request and delay it to the next iteration.

Avoid Carrying Many Defects Forward

As you develop your application, you will find a lot of defects. There will always be pressure from various stakeholders to focus on developing new capabilities rather than fixing existing defects. This is true within a project, since spending time on fixing defects from previous iterations will prevent you from developing new demonstrable features in the next iteration. It is also true that if your next project focuses on fixing defects, you will deliver fewer new features.

Carrying many defects forward will seriously impact your productivity.

Each defect needs to be documented and managed. The more defects you have, the more time you spend triaging them, explaining to people how to work around them, and discussing whether they should be addressed or not. Carrying many defects forward will thus seriously impact your productivity (see Figure 2.6). To avoid wasting tons of time managing them unproductively, fix defects as you find them, rather than carrying them forward. Many teamsor rather, team leaders, customer representatives, or stakeholdersmake the common mistake of not doing this when prioritizing what is to be done. It is the responsibility of the development team to articulate the cost of carrying defects forward, but in the end whether to focus on defects or new features is a business decision.

Figure 2.6. The Cost of Defects.

The more defects you have, the more costly it is to triage them and bear the cost of describing to people how to work around them through support and other means. You should therefore strive to minimize your defect backlog. Fixing defects may temporarily reduce your ability to develop new features but will improve productivity over time.

Comparison with Other Methods

Traditional waterfall development aims to minimize the need for changes through detailed initial specification of requirements and design and thorough subsequent review of those specifications. However, many changes are in fact triggered when stakeholders see partial implementations, rather than through careful review of specifications, and meaningful feedback therefore often comes very late in waterfall projects. Consequently, waterfall development solves problems as they were understood at the beginning of the project. Because change is discouraged, you often have to wait until the next version of the application to incorporate very important changes. Especially when combined with heavy documentation and design approaches, waterfall development often leads either to systems that poorly address user needs, or to expensive cost and schedule overruns.

Many changes are in fact triggered when stakeholders see partial implementations.

Scrum focuses on several techniques consistent with this practice. Scrum proposes iterative development in which the most essential capabilities are implemented and tested in early iterations; at the same time it strongly discourages changes during an iteration (called Sprint in Scrum). Changes are captured in a product backlog and high-priority changes for each iteration are included in the Sprint backlog (compare iteration plan in Unified Process). Scrum, however, lacks distinct phases guiding the team in how to think about what type of changes to make at what time, nor does it include concepts such as Change and Control Boards. Further, Scrum does not provide techniques for how to manage and automate change, understand the impact of change, or deal with refactoring.

XP provides guidance and has helped drive evolution around many of the specific techniques mentioned in this practice, including refactoring, keeping things simple, and avoiding carrying defects forward. Beck describes how these techniques help change the cost equation around change.^[25] Many people have translated this to mean that with XP, the cost of change is constant, allowing changes at any time in a project. Several reviewers have heavily criticized some aspects of this practice. We strongly believe that agile practices can improve the cost equation and enable more types of change to be carried out later in most projects, but it is also important to understand that some types of changes will be increasingly costly for some projects, as outlined in this practice, especially those related to business solutions or architecture.

^[25] See Beck 2004, pp. 5253.

Levels of Adoption

This practice can be adopted at three levels:

• Basic. Ensure that appropriate stakeholders are aware of major change decisions. Use the techniques Partial implementation and Keep documentation, design, and code simple.

  At this level you are reducing overhead through simplicity and facilitating short iterations through partial implementations, taking you down and left on the process map. This is appropriate for all types of projects.

• Intermediate. Use the techniques Effectively manage change requests, Refactoring, and Organize your work to understand change impact. Follow the Unified Process lifecycle to guide you in which changes to encourage and which to assess more carefully.

  At this level you continue introducing agile practices that reduce overhead and facilitate short iterations, taking you down and left on the process map. This is appropriate for all types of projects.

• Advanced. Use the technique Automate change management and change propagation. Minimize changes to the current iteration. Introduce a Change and Control Board to manage more effectively what changes to allow late in the project.

  At the advanced level, you introduce technology and practices that require more discipline and learning. The improved automation enables shorter iterations, while the Change and Control Board has the opposite effect by increasing overhead. The net effect is that this level brings you to the right on the process map.

Related Practices

• Practice 2: Execute Your Project in Iterations describes how you plan iterative development and divide your project into a series of iterations so that you can get feedback on what needs to be changed.

• Practice 10: Prioritize Requirements for Implementation discusses what type of requirements to focus on in the first few iterations of your project. These requirements will force early decisions in key areas to minimize costly changes late in the project.

• Practice 14: Manage Versions describes how to keep track of changes and various versions of assets.

Additional Information

Information in the Unified Process

OpenUP/Basic provides guidance for the basic and intermediate levels of this practice. RUP also adds guidelines for the advanced practices, with guidelines and tool-specific information on configuration-and change-management practices such as parallel and distributed development, as well as information about change control boards and other more formal ways of managing change.

Additional Reading

To understand how the phases in the Unified Process lifecycle help you minimize overall cost of change, see the following:

Per Kroll and Philippe Kruchten. The Rational Unified Process Made Easy: A Practitioner's Guide to the RUP. Addison-Wesley, 2003.

Walker Royce. Software Project Management: A Unified Framework. Addison-Wesley, 1998.

For other practices that help drive down the cost of change, see the following:

Scott W. Ambler and Pramodkumar J. Sadalage. Refactoring Databases: Evolutionary Database Design. Addison-Wesley, 2006.

Kent Beck with Cynthia Andres. Extreme Programming Explained: Embrace Change, Second Edition. Addison-Wesley, 2004.

Douglas W. Bennett. Designing Hard Software: The Essential Tasks. Independent Publishers Group, 1997.

Martin Fowler et al. Refactoring: Improving the Design of Existing Code. Addison-Wesley, 1999.