Letting the Snakes Unravel

XP projects are difficult, more so because the practices are so interdependent. The XP authors appear keen to admit this fact. For example, according to Ron Jeffries,

GROUCHO

I ve observed that when C3 goes off-process, productivity drops substantially. This is consistent with our claim that XP is synergistic. ^[11]

It takes a lot of discipline to keep an XP project on track. Synergistic? Much like knife juggling, it takes a lot of discipline to keep practicing all of the practices all at once, consistently, throughout the many increments of your project. Lose your concentration for just a minute and the tangle of snakes could unravel. If that happens, productivity is likely to drop considerably.

The question is, to what extent can you relax before one of the snakes seizes this opportunity to wriggle loose? The answer depends on which snake wriggled loose.

What If Your Team Has Stopped Writing Unit Tests?

If your team has stopped writing unit tests but is still refactoring away every day, then this will quickly cause problems. Unit tests are a tool for automated regression testing ”that is, testing that a change you ve just made didn t break the existing codebase . By changing the design of existing code, it s pretty much impossible to know whether you ve broken any dependent code without testing it. Without a written design and architecture model to trace dependencies between code modules, you also might not be aware exactly what code calls the module that you re changing. So making these changes without unit tests is virtually suicidal.

How about if a programmer just skips a few unit tests? Say, for example, it s 4:30 PM and you don t want to toss all the code you ve been working on so you can go home clean . . . so you re hurrying to get it integrated . . . you can skip just a couple of unit tests, right? Sure, you can. Then the problem is more subtle, but it would hopefully be discovered before any real damage is done.

An effect observed in XP projects is that test-infected programmers tend to reach homeostasis with their unit tests. They begin by writing too many tests, and then eventually they get lazy, writing fewer and fewer, until the day they notice a bug slipped by. At that point they hike up the number of tests again.

What If Your Team Is No Longer Refactoring?

If your team is no longer refactoring but is also not spending any time producing a design before coding, then the resultant codebase will almost certainly be a botch job of horrific spaghetti code cobbled together with duct tape. Have you seen spaghetti cobbled together with duct tape?

How about if the team is still refactoring a bit but has slackened off because, after all, keeping on refactoring code that the team has already proven to work might become a bit tedious (even for the most rabid of code-sniffers) after, say, a year? What if the team is still refactoring, but not ruthlessly enough to really bash the design into shape? This problem is more insidious than the unit test problem, because failure in this area is more difficult to detect. Bad design is, initially at least, a subjective thing ”that is, until design errors come back to bite you. This usually won t happen until you need to extend the code with new functionality, or when the current increment is released and the customer immediately complains that the program is too flaky, or too slow, or (worst case) crashes a lot under real-world conditions. This sort of problem can, luckily, also be caught early with small, frequent releases; that is, the customer gets to see bugs earlier.

. . . but see the What If Those Small, Frequent Releases Get Less and Less Frequent? section later in this chapter.

The Refactorer s Responsibility

Normally it is the refactorer s responsibility to change all the code that uses the interface that he has changed; therefore, all integrated code must be fully working. However, this does have the side effect (as we discuss in the section What If Programmers Take Ownership of Code? ) that code that was previously thought to be completed suddenly needs to be rewritten, possibly even redesigned, because a separate part of the system has been refactored. As we discussed earlier, many of the XP refactorings involve interface changes, so this is quite a high risk.

Also see the next sidebar, The Snake That Got Away.

The risk of broken interfaces, or interfaces whose semantics may have changed, is mitigated to an extent with extensive unit tests that cover the entire system in exhaustive detail. As long as the programmer runs all the tests, not just the tests for the module he has changed, then the risk is significantly reduced. This is, of course, assuming that the team actually is writing unit tests in sufficiently large numbers and running them regularly. If one snake breaks free, the next one is close on its tail.

Another problem is that none of these practices is free; they all take time. It takes time for the programmers to write the tests, to refactor some code, to discover that a refactoring broke some code elsewhere, to check out the broken code and fix it, to test the fix, and so on.

For a strange but true story about just how time consuming refactoring can be from somebody who has been there and done that, check out the Voice of eXPerience: YAGNI 16 Tests sidebar in Chapter 8.

The Snake That Got Away

During refactoring, the risk of having to totally redesign affected external code is reduced by concentrating on modular design, once and only once (OAOO) and so on. Keeping code modular reduces the amount to which one module affects another module if its internals are changed. In XP, this is achieved through emergent design.

We cover emergent design in Chapter 12.

This does mean, though, that there s actually another slippery snake here to keep at bay. If attention to modular design slips, refactoring becomes more difficult, which in turn makes modular design more difficult to achieve, and so on. If the design becomes steadily more crufty, XP s emergent design process becomes very difficult to follow.

XP s test-first/refactoring approach combined with collective ownership relies heavily on continuous integration to keep the codebase from diverging. However . . .

What If Continuous Integration Becomes Occasional Integration?

XP s recommendation is to integrate the source code as often as possible ”that is, to check working code into the version-control system and do a complete build of the overall system, and then run the unit tests to ensure that all the disparate pieces fit together properly. There s no doubt that a daily integration build is a beneficial aspect to any project, although XP recommends taking this even further, even integrating once per hour . If the team forgets to do this, I doubt you ll see a detrimental effect. However, if the integration rate drops below once per day, then you might well start to see problems.

This is actually true of any project (XP or otherwise ): If programmers have different builds on each PC, getting them all to fit together can prove more time consuming than writing the code in the first place.

This is even more the case in an XP project, where both the codebase and the design are evolving at a fast rate. Many of the refactorings described in Martin Fowler s Refactoring ^[12] involve interface changes. If an interface is refactored on one PC, it s bound to break code on someone else s PC. This may not be discovered until the code is integrated. This problem will be checked to an extent by compilation errors and also by unit tests (which a lot of XP practices rely on). However, compile errors and unit tests don t always catch code divergence early, because the unit tests you re running are the ones that were written for the version of the code that you have on your PC. Therefore, XP requires continuous integration to make sure everyone is refactoring the most up-to-date code. Without it, you quickly end up with ad-hoc versions on different PCs that can become a nightmare to stitch back together.

Writing client code that works one day but breaks the following day because the interfaces changed can also be frustrating, especially when the client programmer simply finds out (rather than being told in advance). So team morale can take a pounding when rapid refactoring is taking place and the programmers aren t integrating their code often enough.

In a non-XP project, code divergence is still a problem, but much less so. Because the design is more stable, refactoring isn t taking place across the whole project at such a breakneck pace, therefore code divergence has less of an impact when it happens. It s definitely still worth doing that daily integration build, though!

What If Those Small, Frequent Releases Get Less and Less Frequent?

If the project has started in earnest with small releases and short iterations, releasing functionality every couple of weeks to a gleeful customer, what happens if the release dates start to slip? (Throwing a bunch of work away at 5:00 PM every day could cause this to happen, by the way.)

It s a likely scenario that as the project gets older, the programmers may become tired of retreading old ground and will start refactoring less. As the code becomes harder to change, gradually fewer and fewer stories will be fulfilled in each iteration, therefore releases will be made less often (i.e., the project velocity will slow down). The extent to which this problem will bite you depends on which other snakes have already escaped and are hiding beneath the furniture. Small, frequent releases give us early feedback, which in turn catches a lot of other issues early. If the releases become too big and/or infrequent, then the snakes will continue to prowl the office undetected, sneakily snarling the project.

Similarly, as the project progresses and people become too comfortable, the iteration lengths might start to increase. An XP team that is switched on to what it is doing will regard this as a big deal, because without fixed iteration lengths it becomes much more difficult to accurately measure the team s progress over time.

What If the Programmers Stop Pairing Up?

What if the programmers stop pairing up? As we saw in Chapter 1, Ron Jeffries sees pair programming as a guideline, an ideal to aspire toward (although other Extremo authors describe pair programming as an unbreakable golden rule). Ironically perhaps, it is not code quality that is likely to suffer directly from a lack of pair programming, but shared knowledge of the project design. Knowledge sharing is one of the primary benefits of pair programming and is one of the reasons XPers feel that they can go for so long without documenting their designs in any detailed form besides source code. If the programmers forget to pair up and, equally important, they do not regularly switch pairs and rotate between tasks , then this communal knowledge no longer flows. This is a real boa constrictor of a problem.

We return to this particular snake in the Pair Programming Illuminated section in Chapter 6.

The extent to which the damage can be limited depends on how tightly knit the team happens to be. If the team members regularly communicate over design issues, are frequently seen to be sketching lines and boxes on whiteboards , and so forth, then the problem should be fairly limited. A worst-case scenario, however, is if there s no communication between the programmers. No one talks to anybody else except about why we don t yet have flying cars , about funny MPEGs of elderly ladies kicking small children, and about how to set up the company foosball league. Project? What project?

In such cases, the problem is more deeply rooted than simply forgetting to pair program. However, this is also the sort of situation where writing things down in detail helps even more than usual.

What If the Programmers Don t All Fit into One Room?

The colocated team is a primary tenet of XP. Without it, you would need a major readjustment of the other XP practices. The on-site customer wouldn t be able to give 100% of her time to all teams , unless she s really a time-and-space-shifting lizard creature from the Ninth Dimension (which is unlikely but not entirely unreasonable).

To enable programmers to work in more than one room (or on different premises, or, taken to an extreme, in different time zones), the requirements would need to be recorded in a more permanent and shareable form than handscribbled story cards because teams in both rooms will need to refer to them (for example, to avoid writing overlapping or incompatible code if the teams are working on similar user stories ^[13] ). In addition, design decisions can no longer readily be shared between teams, unless they do something radical like place all their design documents on a project intranet. ^[14] Of course, this also means that they would have to actually write some design documents and regularly keep them up-to-date as the design evolves. This would be unworkable, unless they also schedule some time to do more prototyping and up-front design, so that the design can first be allowed to stabilize a little bit.

FANGS

So, surprisingly, the simple (and quite likely) scenario of putting teams in more than one room means that we instantly lose the benefits of emergent design, on-site customer, transient story cards, and oral documentation. In fact, for the reasons described, each of these practices suddenly becomes rather risky.

SOLUTION

By patching each of these practices to reduce its risk, we can end up with a software process that is actually a lot more realistic. The refactored process that emerges involves a greater degree of up-front design, early prototyping, design documents, and electronically stored requirements (whether as use cases or user stories, or thou shalt/thou shalt not numbered requirements).

We explore this refactored process further in Chapter 15.

What If Programmers Take Ownership of Code?

At the start of an XP project, each of the XP practices is carried out in earnest. This includes collective ownership, the principle that no one programmer has responsibility for any area of the project. The opposite approach is individual ownership, where each programmer takes responsibility for the code that he wrote.

The industry appears to be divided over these two approaches. Each has its benefits and its pitfalls. In an XP project, however, you really do need to use collective ownership, so that the other XP practices stand a dog s chance in a cattery of working.

Assimilating the Team into the Collective

Just for fun, we did a little research on the subject of collectivism, outside of an XP context. It came from Marxist power to the proletariat sociopolitical theory (more on Marxism later). Here are a couple of interesting quotes that we found:

Karl Marx

Since the supreme aim of collectivism is the abolition of that capitalistic regime which enables one man or one corporation arbitrarily to exploit the labour and the necessities of many men, it obviously does not ”in theory at least ”imply equal compensation for all individuals, nor the destruction of individual initiative, nor the establishment of a bureaucratic despotism. ^[15]

Collectivism holds that the individual is not an end to himself, but is only a tool to serve the ends of the group . ^[16]

Is XP similarly a Borg-like process? The second quote appears to concur with Robert C. Martin s preferences and comfort quote at the start of Chapter 6.

At the root of collectivism is the belief that a collective isn t just a group of individuals, but a single entity. The individuals become secondary to the collective. This could explain why many people see XP as being somehow cult-like.

Karl Marx

Collectivism is a form of anthropomorphism. It attempts to see a group of individuals as having a single identity similar to a person. The collective is claimed to have ideas, and can think. It has purpose, and it acts to achieve goals. It even has a personality, called culture. It claims to have moral rules the collective should follow. It claims to have collective rights, as well. ^[17]

Ask not what XP can do for your project, ask what your project can do for XP.

On the other hand, individual ownership tends to involve specialization of particular areas of code and of particular technologies. Of course, it s human nature to specialize, to carve out a niche and cling to it like a territorial beaver. What if an individual member of the team begins to take ownership over her areas of code? In XP, this is guarded against by pair programming. When two people have bashed out a piece of code between them, it s less likely that one of them will become emotionally attached to the code ( I wrote that, hands off! ). So individual ownership might only begin to manifest if pair programming has also begun to slip. Let one snake escape and the next one is close on its tail.

Individual ownership can actually be a good thing because it creates accountability. It also prevents average (or worse ) programmers from meddling with perfectly okay code. However, in the XP world, individual ownership would cause friction between team members, because there would be a constant fight to keep refactor-happy coders off your base.

On the other hand, collective ownership can, under certain circumstances, have a positive (and appropriately humbling) effect because it forces programmers not to cling emotionally to their code or to take offense if they see someone turning their precious algorithm into something better. This could, however, prove quite frustrating for highly skilled programmers who really do know better than their average colleagues. If you re the one experienced programmer on a team of junior coders, seeing your well-written code get torn apart over time in a relentless display of entropy could drive you to despair. Cue countless confrontations to prevent your code from being rewritten unnecessarily. To top it all, the battle is never won. Come tomorrow, the code that you defended today might have been refactored into an inept programmer s wet dream. Then comes the cry, Okay, who rewrote my code this time?

We can imagine this amusing scenario as follows :

Bang! Bang! I Think We ll Refactor

(Sing to the tune of Maxwell s Silver Hammer by The Beatles)

A pair named Fred and Tim
Checked a bunch of source code in
Then went to the gym

They were very happy with what they d done . . .
Oh oh oh

They d been working hard
Burning through the index cards
And their code smelled clean

They forgot that software is never done
No no no

But Ollie and Stan
Had a very different plan
As we will soon see

They thought that they d have just a little fun
Ho ho ho

And as our boys walked into the gym
Ollie told Stan with a grin Doot dee doo doo

Bang! Bang! I think we ll refactor that code
That they just wrote Doot dee doo doo

Bang! Bang! I think we ll refactor
And they won t get a vote
Just ree-factor
Nobody owns their own code

So collective ownership has benefits, but it isn t the panacea for defective code that it might seem. It just isn t for all teams. In non-XP projects, the benefits of collective ownership can be achieved in different ways. For example, individuals can grow bad coding habits if they program in isolation. This is kept in check with tightly knit teams working closely on related areas of code (but not so closely that they re sitting at the same desk), peer reviews, and close mentoring from more experienced developers. Similarly, bad designs are caught out through design reviews and collaborative design sessions.

What If the XP Coach Falls Asleep?

In pretty much all of the degenerate conditions described in this chapter, the responsibility falls to the XP coach to notice that things are deteriorating and to take steps to correct them. This puts a big responsibility on one person, particularly because most of the conditions described would, by their nature, be recurring.

This heavy reliance on having a good, consistently diligent XP coach is a major reason why XP doesn t scale very well to larger projects. How many good, consistently diligent XP coaches can you find to work full-time on this one project? And worse than that, will they always agree 100% on the sometimes ambiguous definitions of the XP practices?

What If the Cost of Change Curve Isn t Flattened?

Finally, we ask what happens if the cost of change curve isn t flattened. This isn t a practice as such, but it s the desired net effect of all the XP practices put together.

We introduced the cost of change curve in the section The Central Premise of XP in Chapter 1, and we describe it in more detail in Chapter 13.

If the team deviates from the XP practices, the cost of change curve steepens. What happens then? There is, of course, a knock-on effect: Your project starts to get very expensive, and the gold owner s morale will likely plummet, which has been shown to lead to inexplicable termination of the project. This is the king of all the other snakes, the one to avoid at all costs, and yet it s also the one that will inevitably break loose if the other practices fail even slightly.

As we described in Chapter 1, XP supposedly works because it doesn t cost significantly more to add new features in later than it does to add them in now.

Therefore, all of XP s practices must work flat-out in order to keep the cost of change curve flat. All software is subject to the problem of increasing complexity. As more functionality is added, the design increases in complexity to cope with the new features. This increases the cost of making changes later in a project.

XP counters this with aggressive refactoring in an attempt to keep the design of the software simple.

Without constant, aggressive refactoring, an emergent design will consist mostly of regressive mutations . That is, the design won t evolve into a thing of pure beauty but into a banjo-playing redneck with webbed fingers and crooked eyes.

A lot of other practices and principles must be adhered to 100% without fail in order to keep your design pure and good. If you re gunning for an XP-style emergent design, you ll need constant refactoring; therefore, you ll need unit tests and lots of em. You ll also need collective ownership, continuous integration, and pair programmers who are prepared to squeal like a pig if they see a snake on the loose. Is this starting to sound familiar?

So, if even one of XP s practices slips just a little bit, the cost of change increases, and the whole point of XP, the number one purported benefit ”the ability to make changes late in a project in an economically effective way ”is lost.

In this section we explored the dangers of the XP process deteriorating during a project. However, the opposite ”putting the process together in the first place ”is also problematic .

Putting the Snakes Together: Partial XP

How many snakes does it take to complete a circle? Often, teams that say they re doing XP simply aren t. At best, there may be one or two practices that they aren t using. At worst, they might be omitting all of the safety net practices (such as unit testing, pair programming, and collective ownership) but actively employing the dangerous stunt practices, such as emergent design or not writing any design documentation.

Refactoring without unit tests is like skydiving without an emergency parachute . It s cheaper, and quicker when packing your gear in the morning, but when you discover too late that your wife has been cutting parts out of your main chute for her patchwork quilt, you ll really wish that you had taken the extra time.

The following quote was posted to the C2 Wiki by sg :

In the half- dozen projects/places I have tried XP, I have never found anyone willing to adopt the whole thing. I *did* find people interested in trying *one* idea at a time and adding it. For instance, UnitTests. People seem to like UnitTests and will adopt them after being convinced. ^[18]

Often, the problem occurs when XP is first being introduced into an organization. Without proper metrics, and hampered by a slightly kooky name , XP can face an uphill struggle to be accepted. As a result, XP tends to be introduced piecemeal into organizations. The problem then becomes the correct order in which XP must be introduced. Because XP s risk reduction strategy is self-referential, it may be impossible to introduce one or two practices at a time, without accidentally letting a couple of snakes loose around your office.

(Interestingly, about 4 years ago Doug proposed on OTUG ^[19] that it would be quite an interesting combination to merge the front-end use case “driven work he was doing with the back-end coding/testing techniques of XP. He was resolutely rebuffed and informed that all the XP practices had to be used together.)

Our take on this is that there just isn t a correct order in which to introduce the XP practices. However, XPers might then accuse us of being defeatist. Just because something is difficult and high-risk doesn t mean you shouldn t attempt it, right? You can find an example of the possible adoption order that you might try here: http://c2.com/cgi/wiki?ExtremeProgrammingPracticeAdoptionOrder .

Most commonly, automated testing is introduced first, because this has the most immediate and obvious benefits. As soon as managers see evidence of code quality improving, they will (so the theory goes) be hungry for more. Then budding Extremos start to enthusiastically pull their tame snakes out of their bag. This is potentially disastrous, depending on which snake gets pulled out next.

Emergent design is one of the most dangerous practices, because to stand a chance of succeeding, it needs pretty much all of the other XP practices to be in place. However, it s also likely to be one of the first practices to be introduced, because it s the one that many people see as being XP. XP is all about evolutionary prototyping, after all. Even if the programmers involved aren t claiming to be doing XP, they may feel that they re improving their process by introducing emergent design. At worst, this could mean simply not doing any up-front design and taking it from there. However, leaping into production coding without spending sufficient time on requirements gathering and designing, and doing it successfully, requires some pretty labor- intensive safety net practices, all of which must be performed consistently every day without fail.

The danger could be magnified in organizations where XP is brought in as an excuse to avoid design and documentation. In such cases, we would posit that the circle of snakes is rarely if ever completed.

It may seem unfair to criticize a methodology for failing because people don t follow the instructions on the tin. However, more robust software processes have contingency built-in; that is, if people forget to pair program or the on-site customer is on holiday, the whole circle of snakes doesn t unravel.

To put it another way: In theory you might get a completely daisy-chained circle of snakes, but in practice, you d better watch your asp.

^[11] Ron Jeffries posting to the C2 Wiki page All Of Xp, http://c2.com/cgi/wiki?AllOfXp.

^[12] Martin Fowler with Kent Beck, John Brant, William Opdyke, and Don Roberts, Refactoring: Improving the Design of Existing Code (New York, NY: Addison-Wesley, 1999).

^[13] This assumes that somebody cares if you re writing overlapping or incompatible code.

^[14] Some XP teams install and use their own Wiki (see http://c2.com/cgi/wiki?RunningYourOwnWikiFaq) for this purpose.

^[15] Definition from The Catholic Encyclopedia,Volume IV , http://www.newadvent.org/cathen/04106a.htm.

^[16] Definition from Capitalism .org, http://www.capitalism.org/faq/collectivism.htm .

^[17] Definition from Importance Of Philosophy.com, http://www.importanceofphilosophy.com/Evil_Collectivism.html .

^[18] sg posting to the C2 Wiki Page All Of Xp, http://c2.com/cgi/wiki?AllOfXp.

^[19] See http://www.rational.com .