Rejecting the Notion of Doneness

It s worth emphasizing this point, because it seems that anyone who wants to run a software project free of accountability can just reiterate the phrase The schedule is the customer s problem. So that s okay then. Suddenly, the project isn t about getting done but is about developing at a certain velocity. Although consistency and repeatability are important factors in any software development process, they aren t really the goal of the project. The goal (as other XP authors have correctly noted) is working software.

Refactoring the Agile Manifesto

To rewrite one of the Manifesto for Agile Software Development s values ^[4] (with tongue in cheek):

• We value meeting deadlines over constant velocity.

  That is, although there is value in the item on the right, we value the item on the left more. And as somebody once pointed out (thanks, Rob): constant velocity == zero acceleration.

 

This line of thinking ”rejecting the notion of doneness ”pervades the theory behind XP. For example, much of the discussion surrounding C3 s inexplicable cancellation centered around the concept that termination can be success: A project runs through however many iterations and releases, and (if it s successful) it simply keeps going, spinning round, and churning out small releases until the customer eventually pulls the plug.

In XP s favor, if/when the project is suddenly cancelled, it should be in a state where at least what has been written is releasable (depending on how effective the team has been at combating emergent entropy through simple design and refactoring). However, XP teams , simply by following the XP practices, are putting a huge amount of effort into being constantly prepared for the risk that the project might be canned at any moment. The XP life cycle (and all the effort that the team must put into it) is very much centered on the notion that the software is never done, it just runs until it suddenly, without warning, gets inexplicably cancelled.

In the real world things are slightly different. The real world runs on deadlines. For example, in the real world the new product has to be on the shelves in time for the Christmas shopping season , or the new air traffic control system needs to be online by the year 2005, or the payroll has to run if the mainframes drop dead on January 1, 2000.

Imagine if you had spent several million dollars funding a payroll project to protect your company from dead mainframes on January 1, 2000, and then the project manager said to you, The concept of ˜schedule depends on the notion of doneness, which pervades our thinking. You might expect the team to be frog-marched from the building (or quite inexplicably terminated ).

VoXP
Voice of eXPerience: Recovery via Not XP

The November/December 2001 issue of IEEE Software magazine contained a case study of a troubled software project (run by ThoughtWorks, Inc.) that was rescued by XP. In fact, the team saw this as an opportunity to test-drive Extreme Programming in a project recovery setting. ^[5]

Two things are striking about this case study. The first is that the team quickly realized that XP just wasn t suited to rescuing a project with a fixed deadline:

As the severity of the matter quickly became apparent, we were forced to concede that the impending deadline did not allow us the luxury of XP. Instead, we cut up the application and parceled [sic] it out by areas of expertise: servlets, documentation generation, business engines, database, and so on. ^[6]

Despite this, the article was introduced and titled in such a way that it appeared, at first glance, as if XP had saved the project. By the end of the project, the team had not adopted these XP practices: small releases, metaphor, and 40- hour week. The team had only partially adopted these XP practices: planning game, pair programming (30% of the time), on-site customer, and coding standards.

The second striking thing is that (reading between the lines) the team appeared to regard the customer as unreasonable for not wanting to budge over the fixed deadline. Damn that pesky customer for wanting software delivered by a specific date! For example, the team blamed its failure to adopt small releases on the fact that it could never get client buy-in. ^[7]

The fixed deadline was also given as the primary reason for not adopting XP before the software was released:

The client s refusal to budge on the delivery date was the single greatest contributing factor to this outcome. Groundhog Day [the case study author s name for the project deadline] meant that the team could not step back and reassess the situation. ^[8]

However, it wasn t until after the initial deadline had passed and the software had been released (and hence entered maintenance mode, with real live customers to support and train) that the team felt able to begin gradually, painfully adopting the XP practices. By this stage, XP was doing what it is best suited to: maintaining existing systems.

We discuss XP s suitability for maintaining legacy systems in the section When Refactoring Is Useful in Chapter 9.

 

This does raise the issue that XP is simply overdesigned, too high-maintenance in its practices, for developing new systems from scratch. There are more efficient ways of doing this.

 

While Management Gently Weeps

(Sing to the tune of While My Guitar Gently Weeps by The Beatles)

The clock on the wall
Shows the schedule slipping
While management gently weeps

We ve got unit tests
But the design is missing
While management gently weeps

You know that schedule is
The customer s problem
Requirements are too

You just can t let
The schedule control you
Code tells you when it s due

Look at the code
Smell the bugs we are shipping
Still management gently weeps

My pair programmer
Decided to go fishing
Still management gently weeps

Embrace Rampant Scope Creep Regularly

How stable is the release plan? Not at all. The only thing we know for certain about a plan is that development won t go according to it. So release planning happens all the time. Every time the customer changes his mind about the requirements and his priority, this changes the plan. ^[9]

It s not surprising that the XP methods promote scope creep. The whole go home clean, 40-hour week, and refactor til the smell is sweet triad of Extremo philosophy is completely bogus in the real world.

As we will discover in Chapter 13, the Extremos see scope creep as a positively good thing that should be embraced in any project.

 

And without that triad, the only hope of fighting rampant scope creep and endless refactoring is detailed written requirements, schedules with deadlines for deliverables, and up-front design (see Figure 11-1). Fangs is running rampant, because the software engineering practices (written requirements, deadlines, and up-front design) that mitigate risk in the areas of estimation and scheduling have been vaporized in Extremo-land and replaced by aphorisms such as schedule is the customer s problem and the schedule doesn t exist per se.

Figure 11-1. Two very different approaches

The problem isn t with refactoring as a technique. The problem is refactoring in the absence of deadlines, especially because refactoring in XP isn t a scheduled task ”it runs concurrently. It s just a thing that happens all the time and is expected to take up time in each iteration. If refactoring was scheduled as a story (or, more likely, as one or more tasks ), and programmers signed up for it each iteration, then it could be more manageable.

What happens instead is that the amount of refactoring being done affects the project velocity, so a team that diligently refactors every day would be perceived to have a lower velocity because (for a few iterations at least) the team completes fewer stories. Conversely, a team that does very little refactoring might complete more stories for a while, but soon the increasing spaghettiness of the team s code would make it very difficult to implement new stories.

Because of its emergent design approach ( coupled with its dislike of deadlines), XP is a constant fight against emergent entropy, in which its chief weapon is refactoring.

However, refactoring isn t scheduled by the customer. It s up to the programmers to decide how much to refactor. The customer will be told by the team how many stories the team can implement in this iteration. So the decision of how much code quality to add to the product (stir in 3 ounces . . .) isn t a managed decision ”it s a decision that s left to the programmers to make.

Managing Change Through Up-Front Design

XP advises teams not to schedule specific refactoring tasks, but instead to treat refactoring as an integral part of the development process. The benefit of constant refactoring is that design flaws are caught early. Big refactorings are difficult; it s better to catch flaws early and so keep the refactorings small.

As we ve maintained elsewhere, however, spending more time on up-front design (and following a logical design process) helps to drastically reduce the amount of refactoring needed later. Catching design flaws before coding keeps the refactorings almost microscopically small.

SOLUTION  

Because our refactored process doesn t rely so heavily on refactoring, we can afford to schedule it as an engineering task (e.g., a half-day task here or there). This also enables requirements change to be managed, because specific refactoring tasks can be traced back to a particular change in requirements. This makes the process more predictable and controllable, and (non-extreme, though it is) gives control of the team back to the project manager.

Also see the section Use Up-Front Design to Enhance Agility in Chapter 13.

 

 

One analogy is of a news anchor reading from a teleprompter. In his velvetysmooth voice, the anchor reads out the news at his own pace, and the teleprompter speeds up or slows down accordingly . The pace of the news report isn t set by the teleprompter (which supplies the stories) but by the anchor. Similarly, the customer supplies the user stories, but the programmers set both the pace (schedule) and the quality.

We cover refactoring further in Chapter 9.

 

Customers of XP might be surprised to learn, therefore, that quality management is very much in the hands of the programmers. That is, unless they specifically tailor XP to make quality schedulable and therefore estimable .

Quality tends to be the first thing that suffers with tight deadlines, so, without-time on the XP team s side, the practices start to slip and the circle of snakes breaks loose. In Chapter 14, we discuss a case study of an XP project that was conducted by ThoughtWorks, Inc. In this project, the deadlines were tight, so what the programmers sacrificed (whether intentionally or not) was code quality. They didn t have time to clean up the stinky code that did the simplest thing that could possibly work, so they ended up with spaghetti code, and at the end of the project the code was in a worse state than when the team first adopted XP.

Refactoring Iteration

A discussion on the C2 Wiki concerns the concept of a refactoring iteration:

XP requires you to EmbraceChange. Sometimes you have to rip the crap out of your design to meet some changing requirements. When you need to make a fundamental or global design change and some external force beyond anyone s control is driving you to get it done now. You need a RefactoringIteration where refactoring is the theme of the entire iteration. ^[10]

In many ways, we feel that the sentence Sometimes you have to rip the crap out of your design to meet some changing requirements proves our point regarding the benefits of up-front design and detailed requirements elicitation , versus the dangers of emergent design and brief user story exploration. So some XPers (including Don Wells on the VCAPS project) have found that sometimes the design degrades to such a state where they need to stop for a whole iteration and simply refactor ”that s one or more pairs, refactoring, not producing any new functionality, for 1 to 3 weeks.

Not surprisingly, even Kent Beck sees this as a very bad idea (see the referenced discussion page). However, his proposed solution concludes with the characteristically mystical (for Beck) comment:

Whenever we could, we made progress towards our ultimate goal. But we didn t let our fear overcome our desire to serve our customers.

 

VoXP
Voice of eXPerience: eXtreme Road Building

Emergent design isn t just difficult in the software world, as this excerpt from a news item in the Arizona Daily Star shows:

Six years ago, county officials pegged the cost to widen the stretch of Skyline at $10.8 million. On Tuesday, the county Board of Supervisors approved an amended contract for the project that adds another $10.6 million.

The total tab is now $21.4 million.

The cost overrun makes the 2.2-mile road project the most expensive per-mile project in the history of road construction in unincorporated Pima County. Reasons for the cost increase include:

The county s trial of the ˜design-build method of road construction in which contractors design and build the project at once to save time.

County officials didn t know the Skyline widening cost had doubled until recently because the contractor couldn t calculate the entire cost until doing a sizable amount of design. ^[11]

 

But If We Don t Acknowledge the Existence of Deadlines, Nobody Will Force Us to Meet Them, Right?

One of the most important principles in planning for Extreme Programming is that the dates are hard dates, but scope will vary. ^[12]

The variables that govern every software project ”time, effort, scope, and quality ”are handled in XP by fixing time but varying scope (and as we discussed earlier, quality in XP varies in an ad hoc way determined by the programmers). XP uses fixed iterations and releases to the customer often. This can mean that it s very difficult to say specifically, X piece of functionality will be delivered to you by Y date, because the scheduling process is so fluid. Instead, the team can at least say, Some functionality will be delivered to you by Y date, but we re making no promises about exactly what it will be.

If we reverse the equation, we can see that this is roughly equivalent to saying, We will deliver you specifically A, B, and C requirements, but we re making no promises about exactly when they will all be delivered.

To adopt an approach that starts out with the presumption that deadlines are meaningless is like an ostrich thinking no one can see him because his head is stuck in the sand. Interestingly, XP acknowledges the existence of deadlines (e.g., on page 1 of Planning Extreme Programming ), but XP s particular approach to software agility makes it very difficult for the team to actually stick to a preordained schedule. In other words, in XP the schedule is secondary to embracing change (or, if Robert C. Martin is to be believed, the schedule doesn t exist per se ).

Agile planning ”that is, reprioritizing the requirements at the start of each iteration ”is proving useful for certain types of project. As we explore in Chapter 4, for example, XP s arrival was very well timed with the emergence of the dot-com mentality ( We have a business plan, kind of, now let s build us some software! ). In projects where market forces are sufficiently volatile to lead to frequent changes in requirements, or where the survival of the company depends on being first to market, then XP s approach could make more sense than a traditional BDUF approach. However, as we discussed earlier in this chapter, most business projects just don t work that way. Most business projects are driven by fixed deadlines.

This is really the crux of the problem. When the simple concept of deadlines is added to an XP project, look what happens: The team can t refactor indefinitely anymore. So, surprise, surprise, quality suffers. So they d better not have all their code do the simplest thing that could possibly work for the current iteration. So they d better do some detailed, diligent, up-front design. So they d better write down the requirements (in detail ”not just as promises for future conversations).

VoXp
Voice of eXPerience: Estimation and Short Time Frames

by David Van Der Klauw

David worked on an XP project at a company referred to here as CompanyXYZ.

XP concentrates on the extremely short time frame to the exclusion of other time frames. XP says that programmers tend to waste time planning for a future that never occurs. In my opinion, this is a serious fault and a case of throwing out the baby with the bathwater.

Short Time Frame

Spending within a business hinges around payback periods. Whether the spending is dollars or time, the decision to spend or not should always be done on its payback period. A common payback period to use within business is 2 to 3 years.

The short time frame of XP makes the serious error of eliminating many fruitful-tasks that would pay themselves back in a few short months or years. I don t deny that most software development work should pay back in days or weeks. However, the flaw I m highlighting is that XP excludes valuable tasks with a medium or long payback period.

One symptom of the short time frame is that CompanyXYZ is neglecting all learning exercises in favor of completing immediate customer tasks. Many CompanyXYZ programmers feel that their skills are being used up and not replenished. Like a foolish farmer who eats his breeding stock or his seed corn, CompanyXYZ isn t allowing its programmers to take time to learn anything that doesn t have an immediate payoff to a customer.

In business, there are many techniques that will give a short-term boost to productivity but with long-term damage. These techniques are often pushed by consultants or managers who can come in, quickly show a short- term gain, cash out, and then leave the problems behind for someone else. I believe that XP is one of these quick-fix fads.

The short-term emphasis leads to low quality of coding. There are many altruistic reasons for writing quality code, but one benefit of the conventional code ownership is that it forces you to write good quality code so that you won t be stuck with a mess in a few months or years time. With conventional code ownership, if you failed to plan ahead, you would be dragged back in to fix the design. With XP, however, any problems down the track are not your problem. In practice, this results in less care and lower quality results.

For a discussion of what can go wrong with collective ownership, see the section What If Programmers Take Ownership of Code? in Chapter 3.

 

Estimation and Spikes

Estimating the time a job will take is one of the most difficult aspects of software engineering. XP attempts to solve this problem by demanding accurate estimates for routine work and requiring investigative spikes for tasks that cannot be accurately estimated. I believe this is a serious flaw in XP.

Estimating software is so hard to do because, unless you are a complete idiot, every job involves something you have never done before. After all, if you have done it before, a copy and paste from your archive will take only seconds.

XP pretends that work can be conveniently divided into routine work that can easily be accurately estimated (yet too hard for a single person to do by herself) and groundbreaking work that will always require doing an investigative spike, then throwing away code, then estimating, then writing tests, and finally performing the task. If only life were so simple.

Perhaps your car s accelerator could be replaced with two speed buttons : one speed for highway cruising and one speed for local roads . In reality, there is a full spectrum of routine/experimental work that is experienced throughout just about every task, just as your car experiences a range of speed on every journey.

For example, as I code a FOR loop it might be 100% routine. I use a familiar API function but in a new way: 50% experimental and 50% routine. Then I try to cast the return to an unfamiliar type and it is 100% experimental.

The distinction between spikes and production code is not a useful distinction. The two are heavily entwined and cannot be separated.

I am not saying that there is no value in making an estimate and reevaluating when the time is up. Estimating is worthwhile, but it will never be accurate. Trying to make it so with the worthless distinction of a spike is futile.

I am not saying that there is no place for dedicated spikes, nor am I saying that work is never 100% routine. What I am saying is that a great deal of work cannot sensibly be divided into the categories of experimental and routine, because it is a mixture of both. Forcing this distinction is a flaw in XP.

Estimation in Practice

XP involves breaking down every task into very small sections and estimating the time needed very accurately. Should an accurate estimate not be possible, an investigative spike is first done, then an accurate estimate is made, and then the task is started.

In practice this does not work. It was interesting to see the various ways our developers defeated this system:

• Some simply did the work, and after completion they did a task breakdown and entered the time actually taken (as their estimate).

• Some wrote the first thing that came to mind, estimated it to take 4 hours, and then started the task. If it took longer, then they simply added further tasks of 4 hours until the job was done.

• Some gave a generous 4-hour spike for each task. In this time they performed the task and then allocated half an hour to finalize their work.

 

^[4] See the Manifesto for Agile Software Development at http://www.agilemanifesto.org.

^[5] Peter Schuh, Recovery, Redemption, and Extreme Programming, IEEE Software magazine, November/December 2001, p. 2.

^[6] Ibid.

^[7] Ibid., p. 5.

^[8] Ibid., p. 4.

^[9] Kent Beck and Martin Fowler, Planning Extreme Programming (New York, NY: Addison-Wesley, 2000), p. 41.

^[10] See http://c2.com/cgi/wiki?RefactoringIteration.

^[11] Tony Davis, Widening cost double for Skyline, Arizona Daily Star , February 5, 2003.

^[12] Kent Beck and Martin Fowler, Planning Extreme Programming , op. cit., p. 84.