Building an Infrastructure with Emergent Design | Extreme Programming Refactored: The Case Against XP

GROUCHO

What about that database you need to build first? What about that framework? What about that syntaxdirected command compiler? Get over it! ^[5]

Chapter 26 of Extreme Programming Installed (we ll call it XPI26 for now, because we refer to it quite a lot) gives a fairly succinct description of the theory behind emergent design. XPI26 also mentions and reinforces the Extremo yell of YAGNI!

Similarly, XPI26 pulls no punches in the message that it communicates (we re talking here about transitioning the developer mind-set from the more traditional up-front design, or look before you leap approach, to emergent design):

As you go along, put more and more simplicity in at the beginning, relying on your ability to refactor to add the generality ”or the general tools ”that you ll need. ^[6]

The concept of putting simplicity in struck us as a conceptually backward thing to do ”rather like opening a window to let the cold in rather than let the heat out. Perhaps it s a Zen thing. Regardless, the emergent design approach is backed up by XP author Martin Fowler, who (talking about the joys of refactoring) said the following in an interview:

I let most of the design flow from the evolutionary process. So I feel that there s been a shift in balance. Before, I might have preferred ”and these percentages are purely illustrative ”80% of my design in planned mode and 20% of it as the project went on. Now I d perhaps reverse those percentages. ^[7]

This is a startling ratio, illustrative or otherwise . The implication is that very little time is spent on up-front design ”less than a quarter of the overall design, in fact. Ron Jeffries (quoted at the beginning of this chapter) takes this to even further extremes by stating that 10 minutes of up-front design should be sufficient.

This approach might just work on very small projects, where 10 minutes of thinking is likely to generate a workable design. However, on complex projects, on the bleeding edge, it might take a lot longer to get in touch with reality, no matter how much we d like it to be otherwise.

Here s another quote that made us scratch our heads:

GROUCHO

Emergent architecture relies on looking at a solution with a poor architecture and making a better architecture. ^[8]

We wonder , why deliberately start with a poor architecture? What s the benefit-of starting with a poor architecture? Wouldn t we want to start with the best architecture that we can?

Why XP Deliberately Starts with a Poor Architecture

The philosophy behind XP s emergent design approach is somewhere along these lines: We start with a poor architecture because we don t know yet what the best architecture is; we learn only by doing; and maybe the first-pass architecture will suffice after all, and if so, we haven t wasted time in speculation. Furthermore, in the Test ˜ Code ˜ Refactor cycle, you do, according to Extremo theory, end up with the best architecture that you can, and it has taken, if all goes well, less time than Prototype ˜ Analyze ˜ Design ˜ Code.

This philosophy is based on the (erroneous) assumption that you can t get the architecture and design right by designing up front. In this book, we re making the case that by following a logical design process and by prototyping during designing, it s possible to get the design a lot more right than Extremo wisdom would suggest.

See later in this chapter for more about the benefits of combining prototyping with up-front design. Also see the section Is Up-Front Design Sufficient to Avoid Large Refactorings Later? in Chapter 9 for more about the benefits of applying a logical up-front design process.

Using XP s approach, you don t set out to design a good architecture first and then code it. You just leap into the code with minimal up-front design (as described by Ron Jeffries in XPI26) and wait for the architecture to emerge. Spending time thinking about the design first is denigrated by the Extremos as Big Design Up Front (BDUF), another XP design mantra.

We discuss BDUF further in Chapter 8.

On first scan, XP s design approach does make sense. Designing for something that you might not need is often a bad idea, and leaving that part of the design until later can definitely save time, even if it turns out you do need to add that something in after all. However, XP once again takes things to extremes, and this is where the idea of emergent design (in the XP sense, at least) goes awry. We cover the reasons why in the remainder of this chapter.

Frameworks: What If Your Code Has Design Value Instead of Business Value?

In XP, the disparagement of up-front design extends to framework design and implementation (i.e., spending extra time putting code infrastructure in place). You might end up with a framework (of sorts) at the end of an XP project, but you certainly don t start out by writing or designing a framework.

This approach is described in XPI26. The driving force is that all the code you write should provide immediate business value to the customer. All code must start off with a user story, which the customer will have chosen to be developed next . Frameworks tend to be behind-the-scenes code ”the underpinnings ”so they aren t directly attributable to business value.

If you need to create a nonvisual component, which doesn t itself provide immediate business value (perhaps it provides support for some other component), then of course it needs a user story of its own. The user story should describe the new component so that the customer thinks, Gosh, I really do want one of those after all!

The same goes for generic code that applies to more than one user story but doesn t itself qualify as a user story. Thus, it would be nearly impossible to justify starting a parallel project (however small in scale ”for instance, a week s work) to produce some common code that can be shared by different modules or even shared between projects.

The stumbling block is that anything that doesn t provide immediate business value to the customer (i.e., that doesn t get written as a user story) can t be justified. Therefore it s considered to be low priority and gets pushed to the back of the queue or dropped altogether. If you think about it, this means that sometimes the customer must think like a designer in order to justify whether a user story is worth including in the next iteration.

There are two issues to explore here:

The customer must sometimes make design decisions (despite the fact that the XP literature preaches otherwise). Often, even minimal infrastructure code can make a program simpler and reduce development time. This sort of code provides design value rather than business value. However, for such code to be written, it would need to be written up as a user story and signed off first by the customer. This may seem contrary to XP theory, but we will attempt to explain why in practice this would turn out to be a customer decision.
An up-front design would make planning (therefore prioritization of features) more predictable. If the infrastructure code is quite small scale, it could be refactored into existence without the need for a user story, but as the infrastructure gets bigger, project velocity will drop. From the customer s point of view, the software is just taking a lot longer to write than had been anticipated, but with no real feedback on why. Instead, more code is being produced that doesn t relate directly back to any particular user story. An up-front design would have allowed the team to better predict what needs to be written and how long it would take ”cutting a straight line from A to B.

It s worth walking through a brief design example to illustrate both points.

Example: A Services Framework (Nothing to Do with Web Services!)

Suppose your project consists of many interrelated GUI dialogs that link to common services behind the scenes. Services might be for anything: printing, linking to a remote EAI hub, connecting to a banking system to transfer funds, and so on. These services communicate with and delegate to each other to provide responses to GUI events.

The simplest possible design would be for each service to know exactly which other service it needs to delegate to in order to get the job done. This is great, and it should give us a clear and simple design ”at first. However, as the system grows in complexity, we quickly discover that it s difficult to add or extend services, because each caller (the class that invokes a service) must be made aware of the new or changed services as well.

Not Much Functionality at First: Nice and Simple

Let s say that our mythical application is a content management system in which the static parts of the Web site are designed offline and then uploaded to the public Web server.

To facilitate this, we would need a service that connects to the Web server in order to upload the static files (HTML files, images, etc.). This Web Server Upload Service is in turn called by a service that is responsible for publishing all recently modified files for aWeb site. This Web Publisher Service is in turn triggered directly from the GUI. Figure 12-2 shows the basic class structure.

Figure 12-2. The basic class structure, early in the project

Adding More Services: A Bit More Complex Now

Now let s say that the customer has identified a need for a new upload mechanism to upload via a secure connection. This Secure Web Server Upload Service extends our existing WebServerUploadService class. This is fine ”a simple if statement in the Web Publisher Service will determine which upload service to call, depending on a global user setting. Now let s say we want to add another upload type (an FTP upload, for instance) so our if...then statement becomes a little more complex. Figure 12-3 shows the new class structure.

Figure 12-3: The class structure a bit later in the project not quite so neat and tidy now

The WebPublisherService class is now a potential minefield of complexity ” and all because we wanted to keep things simple. Now every time we want to add a new upload service, we need to modify WebPublisherService to make it aware of the new service to call.

The obvious solution is to make the choice of service implementation transparent-to the WebPublisherService class ”in other words, that all WebPublisherService knows about is the FileUploadService interface. Now, however, things get a little more complicated. WebPublisherService itself needs to know which service to delegate to, without knowing what those services are. And we especially want to avoid the very non-object-oriented situation of a stream of if...then...else statements to determine which service to call.

As the overall system gains complexity, it s likely that the exact same problem-is cropping up in many different parts of the system. We need a proper way of extending the system with new services, without breaking existing code ” ideally so that the caller can t tell that it s using a different service.

Adding a Little Complexity to Make the Solution Simpler

One solution would be to introduce a new back-end module ”a generic mechanism for looking up services, say. Of course, we can t justify this extra module without a user story, so XPers should just skip to the end of the chapter now. . . .

In fact, it would be pretty much impossible to write a genuine user story that asks for a design feature. (Of course, we could write such a user story that purely adds design value, but then ”hey, guess what, the customer is now thinking like a designer!)

Anyway, back to the example. Using our generic lookup mechanism, the caller would simply pass in the details of the type of service (e.g., the interface of the required service) that it wants and be handed back a concrete implementation of that service. So to round off this example, the caller could pass in a service interface called FileUploadService and receive back one of the implementation classes (e.g., WebServerUploadService), depending on the user configuration. The caller is none the wiser about which implementation it receives ”all it needs is the FileUploadService interface. The resultant class diagram would look something like the one in Figure 12-4.

Figure 12-4: The class structure with some added infrastructure. Ah, that's better!

The addition of a little extra infrastructure would almost certainly provide savings in terms of time spent on development, because it results in a much cleaner overall design. But it would need time spent up front creating it, without any appreciable new functionality appearing. In fact, it might take an entire 1-week iteration to code the basic ServiceLookup class and its associated support code. ^[9]

And Then the Rest of the Design Falls into Place “Isn t It Great When That Happens?

With this infrastructure in place, we begin to see other benefits emerge ”and before any code has been written, too!

For example, each service would need to handle its life cycle (i.e., starting and stopping, and multiple instances ”for connecting to different servers, for example). Using XP s emergent design technique, life cycle management would be added to individual services, initially the service for which the need is first identified, then the next one, and so on. This would continue until after a while somebody notices that each service is doing something similar (life cycle management) and the design could be simplified a lot by having some sort of centralized service manager that handles life cycles for all the services in a consistent way (e.g., calling startup() and shutdown() on each service object at the correct time, and handling failure modes). It s time to grab our pairprogramming buddy, refill the snack bowl, and start refactoring!

The trouble is, having evolved independently of each other, each service would be handling its life cycle in a slightly different way, probably having been written by different people (but we re forgetting this is the Borg collective, and everyone here thinks alike).

This inevitable (and subtle) divergence of the code can make refactoring much stickier. This sort of problem ”unstitching similar, but not altogether similar, code ”can make refactoring a progressive, time-consuming chore. Each individual service must be re- analyzed (there s no logical design to refer back to, after all), and each service s start/stop behavior must be picked apart and all its semantics deciphered so that we don t accidentally remove any essential behavior by commonalizing the code.

Meanwhile, if we had simply taken the up-front design approach and been able to put our Service Lookup infrastructure in place, all this would have been solved already. By assigning behaviors to the classes in Figure 12-4 (possibly via a sequence diagram), we would have identified almost immediately that each service will have startup() and shutdown() methods (plus a need to exist as more than one instance).

There is obviously some common behavior here, so it makes sense for some of this code to go into a new superclass called AbstractService and for the life cycle management to be controlled centrally by the ServiceLookup class, which basically becomes a Service Factory (to use the Factory design pattern).

By identifying this right at the start, the design has been kept simple before any code has been written ”before the code has even had a chance to diverge or for duplication to appear. So . . . no need for all that time-consuming refactoring!

But How Do We Justify This to the Customer?

In an XP project, this extra infrastructure might never be added because it isn t directly attributed to a user story . The Service Lookup infrastructure doesn t provide direct business value to the customer, so we can t justify allocating resources to code it.

Therefore, we ll just have to rely on the code gradually, painstakingly being refactored into something similar in perhaps a year s time (with all the associated pain of a bad design in the meantime), when it is blatantly obvious, from looking at the design model, that spending time coding this up front would save a lot of time, even in the short term .

In a non-XP project, no such restriction exists: The customer defines the requirements, and the programmers design the solution. This is supposedly the same in XP (in fact, division of responsibility is central to XP rhetoric). However, as we ve demonstrated, sometimes the customer sits a little too close to the programmers and may be required to make design decisions in order to keep the XP flag flying.

And as we discussed in Chapter 5, the customer is already overloaded with responsibilities without having to become a software designer as well.

It might be possible to add a user story that says, Write something that gets me the finished system several months sooner, and then the programmers can implement their Lookup mechanism. But then we enter the realm of controlling the planning game with meta user stories, which XP certainly doesn t cover.

We re not suggesting that these problems are insurmountable in an XP project. In the kind of scenario we just described, any development team with a collective ounce of common sense would realize that they need to spend at least a portion of their time producing code that isn t directly attributable to customer value (at least not without some pretty contrived user stories).

In such cases, the most likely course of action is that the team will estimate the amount of time it will take to produce the additional infrastructure and schedule it as one or more rogue user stories (i.e., ones that don t have immediate business value but instead add design value). However, in this case it s important to understand that the team is no longer performing XP. The team has strayed from the path , however deliberately. It is effectively tailoring the process: producing a customized version of XP that s adapted to the needs of its own project.

However, tailoring XP is actually much riskier than it at first seems. The circle of snakes is already complex and insidious without adding our own customizations, and it s all too easy to let the snakes unravel.

Using our circle of snakes metaphor, we describe the risks involved in tailoring XP in Chapter 3.

From A to B in a Straight Line

A key point to the services example is that the design was evolved and refactored before a single line of code was written.

The equivalent (taking an emergent design approach) would have involved writing unit tests, writing some code, realizing the code was wrong, changing it, fixing it because the tests caught errors introduced by the change, adding more tests, changing the code again, and so forth. Test-first design ^[10] is useful when used in conjunction with up-front design (because they address different aspects of the design), but on its own, emergent design is too high-discipline and not as efficient as it should be.

We discuss test-first design in Chapter 8.

Up-front design is agile in the sense that nothing we write down is fixed. Diagrams are, simply, easier to change and update than source code.

Up-front design (done properly, of course) isn t speculation; it s thinking and planning ahead. As such, up-front design provides us with a roadmap. Having produced the design, we don t need to implement everything in it right away. Instead, we can prioritize engineering tasks, where each task traces back to one or more customer requirements. (Note that this isn t dissimilar to XP s approach, with user stories broken down into tasks and stories penciled in for each iteration.)

The services example shows us that a services lookup framework isn t particularly complex. For example, we may not need database persistence (yet). With emergent design, we would assume at every stage that we need only the simplest possible design for now, even if we need to rewrite it later.

Conversely, with an up-front design, it s possible to make a much more intelligent decision about whether to code something simple for the time being, part of which might need to be rewritten when we implement the rest of the design. It all rests on if the customer really needs this simpler functionality sooner, or if he would prefer to wait and get more functionality delivered in a shorter time frame (with less rewriting).

Time to market might appear to be everything, but this isn t always the case. Often it pays to spend more time getting the product right first.

Keeping the Design Agile

SOLUTION

Looking at the UML model in Figure 12-4, it becomes obvious that passing File objects back and forth isn t a particularly abstracted solution. The java.io.File class is essentially a handle to a file system resource, so this ties the design into a files-only solution. What if we want to upload something that isn t a file? There could, for example, be a concept of a virtual file-system (e.g., hierarchical configuration details).

One possibility would be to replace the File parameter with an InputStream. This would cover pretty much all eventualities without adding any real complexity.

This is where up-front design (especially when combined with prototyping) has great advantages over emergent design. We get to see patterns emerge and consolidate the design into something both simpler and more generic before any production code has been written.

Is this the same as guessing ahead to future requirements? Of course the answer to this will vary. The amount of guesswork depends partly on the volatility of the requirements and also on how far ahead you re trying to design. The skill is in not guessing, but in learning to identify real requirements based on real problems that the customer needs to have solved ”then to build a high-level design around those requirements ”and then to prioritize implementation of the design into short iterations, which are in turn designed up front in more detail.

Up-front design helps us to quickly massage the design into something that is modular and therefore malleable. And because this needn t take very long, it helps the project to be more agile. If the requirements change or new features are added, the documented design helps us to quickly identify dependencies before any code is written.

Up-Front Design Is Not ˜ Big Delay Up Front

It s important to make this point. Producing a comprehensive design up front, done properly, doesn t have to take a long time.

SOLUTION

The extreme that we all want to avoid is the one where no code is written for a year or more because the architects are still perfecting the design documents. The ideal middle ground is where the design takes one or two iterations (in XP time ) ”maybe about a month, including prototyping. Of course, this is still a lot longer than XP s recommendation of 10 minutes of design up front.

Starting with the high-level architecture also helps, because individuals (or individual teams ) can then get started on different areas of the detailed design in parallel. Coding can also begin on individual subsystems before the big design is finished, as long as the team has a very clear idea of what it is doing.

Meanwhile, testers can get started writing test scripts from the requirements, and programmers can write prototypes that are immediately fed back into the design. This all helps to make the design more stable, and it ultimately saves a lot of time and effort (we discuss this in more detail in the next section).

Is it really that important to give the customer working code a couple of weeks into the project? If the customer is prepared to wait just a few weeks extra, she will get a much more robust design and functionality ”delivered incrementally ”that gets delivered faster and is less buggy .

Going into more detail about the design earlier can also help the team to produce a more accurate set of estimates earlier ”that is, the team can better predict, at an earlier stage, when the project will be completed. ^[11] The team can do this because the design process involves breaking down the requirements (e.g., use cases) into schedulable, estimable engineering tasks.

SATIRE WARNING
eXtreme Building (XB)

by Mark Collins-Cope

There s a new construction (of the building type) method out called XB ” eXtreme Building. Basically, a lot of brikkies (bricklayers) grew frustrated that architects, civil engineers , electrical planners, etc. earned a lot more money than they did, and they decided that they were a redundant waste of space. So XB was born.

With XB, buildings can be put up in much less time. XB teams consist of a small number of talented brikkies who go on site, discuss what they re going to build informally (over a cup of tea in the tea hut), and then get to it.

Initial results are amazing. One team managed to construct a small 10 10 10 room in less than a day. They re now starting to think about what the rest of the house is going to look like, but initial indications are good, and the building team is very happy with its initial display of progress.

One recent innovation is refactoring the building. It turned out that the 10 10 10 room had the door in the wrong place (initial thoughts on the position of the second room were changed). But this wasn t a problem. They refactored the room in no time. Basically, every brick in the new door position was meticulously moved to the old door, and voila! Everything was all right.

Unfortunately, this took longer than it took to build the initial room, but heck, at least the construction manager had something concrete to show his clients on day one!

Plans are now afoot to build the first XB skyscraper (or is it a bridge?). Anyway, more news as it arrives. . . .

^[5] Ron Jeffries et al., Extreme Programming Installed , op. cit., p. 189.

^[6] Ibid., p. 192.

^[7] Bill Venners, Flexibility and Complexity: A Conversation with Martin Fowler, Part IV, http://www.artima.com/intv/flexplex.html, November 5, 2002.

^[8] Kent Beck posting to the C2 Wiki page Can An Architecture Emerge, http://c2.com/cgi/wiki?CanAnArchitectureEmerge.

^[9] Well, okay, for this example we could get away with a simple HashMap, but this is a simplified example, right?

^[10] A key component of emergent design.

^[11] Note that this is still very much subject to the usual uncertainties that afflict many software projects, but with a decent design process, these uncertainties are more easily controlled. In fact, ignoring the benefits of detailed up-front design because design is difficult to get right early on is (to use a cliche) throwing the baby out with the bathwater.