Chapter 7. Continual Refinement

In order to achieve sustainable development, teams need a way to balance short-term requirements and long-term needs: to ship their product as soon as they can while accepting, anticipating, and even welcoming change.

Software is complex and must evolve continuously over a long period of time. Throughout its life cycle, a software project must contend with changing user requirements in an evolving ecosystem of markets, partners, technology, and competition. In this environment, just about the only thing a software team can rely on is that change is a given. While some changes can be anticipated, teams must accept that most cannot; there are no crystal balls to predict the future. Experience with a particular software ecosystem can help, but it is still no guarantee for success. Therefore, teams must adopt a mindset and related practices that help them easily adapt to change as it happens while also anticipating change. The core agile software development practice of iterative development (described below) encourages continual refinement because it is a lightweight but formalized practice that helps teams adapt to and anticipate change.

In agile development, teams work from a simple and clear vision and deal with change through frequent iterations; the goal of each iteration is to deliver something useful to customers. Iterations begin with the team's laying a plan for upcoming iterations but with a focus on the next iteration. This planning takes into account the most recent available information such as an understanding of what the team has accomplished in past iterations (velocity, described below) customer feedback, or technology changes. The team then executes on the plan for the duration of the iteration, usually just a few weeks, delivers the software to the customer, and then takes a few moments to reflect and learn how it can further improve before planning and starting the next iteration. The agile methods keep the process as lightweight as possible so that the focus is placed on achieving results, while the tight feedback loop with customers encourages teams to do only what is most useful to the customers, which helps minimize wasted effort.

The main advantage of agile methods is that they help teams manage uncertainty and change early and throughout the project. However, while all agile projects combine aspects of anticipation and adaptation, poorly implemented agile methods can result in sloppy planning and reactive thinking [Highsmith 2004b]. Sloppy planning results when teams ignore information they already know or should know with minimal effort. Reactive thinking results when too much emphasis is placed on the team's ability to adapt to change. The result could be a project where lots of changes are being made but the amount of actual progress is low. Hence, the challenge with agile methods is to rely on both adaptability and anticipation and to have a team that knows when and how to apply each, and in what measure.

Traditional methods of software development rely too heavily on upfront planning and anticipation and virtually ignore adaptation. These approaches stress attempting to understand all user requirements and producing a detailed design before writing any software.

Agile methods are not the exact inverse of traditional methods; the common agile statement "we have more confidence in our ability to adapt than our ability to predict the future" should not be an excuse to ignore anticipation. Instead, in agile methods there must be an understanding of good enough anticipation. For example, spending a few days at the beginning of a project understanding user requirements or spending an hour every day discussing software design are worthy activities because they can cut down on rework and even, if done right, increase the ability of the team to adapt to change. The latter topic, especially as it applies to design, is dealt with in greater detail in Chapter 6 on Design Emphasis.

The need to involve both adaptation and anticipation in agile methods is similar to the need to consider both the short and long terms. In software projects, if too much emphasis is placed on the short-term, then software quality will suffer because the team will be continually reacting to crises and not being proactive. This will result in a continual degradation of the ability to adapt to change as described in Chapter 2. Equally as dangerous is when too much emphasis is placed on the long term (i.e., anticipation); these are the projects that are continually at risk of never shipping because they are unable to deal with changing user requirements through their emphasis on planning and design.

Agile Practices Are Still Useful When Adaptation Is Not Required

There are some projects where agility seems less important, where the temptation exists to stick with traditional approaches. Examples are where a software defect might lead to a loss of life or injury or the failure of an expensive piece of equipment such as in medical or aerospace applications, or in projects where milestones are tightly defined and must be approached in a given order. However, even in these projects there is value in lightweight collaborative planning and project tracking as described in this chapter, because virtually all projects can benefit from the steady rhythm and discipline of regular planning and tracking available in iterative development. Teams don't need to make changes to their plan in every iteration, but having the ability to change if needed could be the difference between a successful or failed project.

In my pre-agile days I was a believer in Microsoft Project and its Gantt charts. However, despite repeated attempts, I never produced a project schedule that stayed valid for more than a few weeks. If I didn't make an effort to create a new schedule every few weeks, then I would proceed blind and trust my instincts, and because creating a new schedule was so much work, I was probably flying blind more often than not. This is where a lightweight planning and tracking method as advocated by agile development would have been extremely useful.