Agile Outside Software Development

Agile software development calls for customer collaboration. Collaborating actively with the customer will immediately bring into discussion the company's contracts, sales, maintenance, and communications relationshipsin other words, the whole business. A company that takes the agile model seriously will have to make adjustments in all those areas.

Fortunately, the agile approach works well in most business environments. Let us look at a series of cases starting from the software arena and moving outward to business in general.

Project Portfolio Management

Any large organization has a portfolio of projects underway at one time. The management team should balance the effort being spent on each project on a quarterly basis, if not more often. They run into two problems: staff specialization and finding stopping points.

In most companies, one person becomes the expert on one piece of code. Every time a function is changed in that piece of code, the organization is dependent on that one person to do the work.

One of the characteristics of agile development is increased communication, hopefully with Osmotic Communication. In some cases, as with XP shops, programmers work in pairs and change programming partners frequently. This means that more people learn to work in each part of the system. With rotating pair-programming partners, the organization can shift a programmer to work on a more critical project when needed, with (possibly less qualified) people able to work on the other code.

Even without pair programming, it is still a good idea to cross-train people on each other's code bases. The understudy may be less proficient but will still be able to carry out the needed work.

The second problem with project portfolio management should be easier to deal with. With properly executed agile development, the system is put into a stable, shippable state every week, month, or quarter. At those points, the people on the project can be moved to other projects as needed. There is a danger of thrashingmoving people back and forth so often that they lose track of, or interest in, what they are working onbut there is at least the option of rebalancing the project portfolio to reflect the organization's current priorities.

With properly executed agile development, project teams will be working on the most valuable features of each system^[43] in each increment. This means that when a project is trimmed or put on hold, it is always in a usable state. It might happen that the organization never restarts the project or changes its direction. These are both valid outcomes of project portfolio management.

^[43] See the discussion of work-sequencing strategies (Cockburn CC 2005a, p. 48) and the use of the "iceberg list" (<alistair.cockburn.us/index.php/Earned-value_and_burn_charts> June 22, 2004) for a more complete discussion of this idea.

There is, of course, one remaining problem: the difficulty of obtaining meaningful market data to decide which features to move forward into early increments and which to move back or drop. I have not yet met any market analysts who can provide comparative market forecasts for products down to the feature level. Real project portfolio management requires that the management team be able to compare cost and benefit numbers at the feature level in order to decide how to best balance their use of scarce resources in development.

It probably will take a combined group of senior developers (to provide the cost numbers at the feature level), market specialists (to provide the return numbers at the feature level), and creative management facilitation to engage the group in discovery of interesting strategies for development sequences in order to come up with good answers to the project portfolio question. Note, as before, the importance of keeping the cooperative game and the "there's only us" mind-set.

Customer Relations

Companies that do custom development may have trouble with agile development because their relationship with their customers doesn't support the point of view that "there is no them; everyone is us." When the contracted company proposes that the customer visit the project weekly to provide input and steer the priorities on the project, the customer replies, "But that's what we hired you to do!" This is a lose-lose worldview. The customer doesn't get the best system they can, and the provider adds risk to the project.

Experienced readers will recognize that the agile model of customer relationship matches that coming from Toyota and lean development (Ohno 1988). One would therefore hope that close customer collaboration would be easy to achieve. However, my reading of the lean literature doesn't make me very optimistic on this score. Toyota's customer- and vendor-relation model has not penetrated the minds of most executives even after decades of publicity and exceptional results. I can only think that every additional push from the agile side helps. Tomax's results (see next shaded box) are heartening because they show that customers can become used to, and then expect, to participate in steering their projects.

Contracts

Companies that do work on contract are always faced with the problem of price, scope, and time (this applies to all industries). Clients often want all three to be fixed to reduce their own risk. I see (to date) only a few options.

Fixed-Price, Fixed-Scope (and Possibly Fixed-Time)

This option is the old one: Do your best, however you do it, to come up with a number for the project. Once inside the project, squeeze all the fat out of the process, using all the best agile techniques you can muster (see "Process, the 4th Dimension" [Cockburn 2003c]) to deliver it as well as you can. As far as I can tell, agile techniques do not change your ability to estimate a project. Despite our most fervent wishes, estimating a project is still a matter of previous experience, feel, and guesswork. Fixed-everything projects make sense when the requirements are very stable. They are also used when the two parties don't trust each other, even though there is usually plenty of leeway for one party to twist the situation to the other's disadvantage if they want.

Fixed-Price, Fixed-Scope (and Possibly Fixed-Time) But Collaborate with the Customers to Alter Scope Anyway

A risky approach. I have seen it done successfully several times, twice for commercial projects and once for a state government project. Jeff Patton (Patton 2005) describes working closely with the customers at the start of the project to identify needs, wishes, and priorities. As the project proceeded, Jeff made sure they were always working on the most important items. As they got close to the end of the time period and it was clear they would come up short on features, he made sure that the items left undone were clearly the least important ones. The customer either forgave the absence of those features or rolled them into a follow-on contract.

The government contract I tracked was again a fixed-everything contract. The team worked closely with the users of the system, delivering and installing running, tested, usable software every month. After each delivery, they discussed with the users whether to continue with the contract as written or change directions based on what the users really needed. I found it remarkably brave of the contracted company to follow the users' requests in deviation because in a hostile sponsorship situation, they would have been at risk of non-payment. However, the user base was so happy with the usefulness of the functionality delivered that the contracted company got paid in full. This was a case of converting a fixed-price, fixed-scope, fixed-time contract into a time-and-materials contract on the fly. A dangerous but wonderful strategy if you can accomplish it.

Time and Materials

This is simply a matter of paying for work as it gets done. It is obviously the best format to use if the requirements are volatile. In one caseI won't even call it a projectthe sponsor changed the requirements wildly, even up to days before delivery. The contracted company always delivered whatever they had running at the end of the quarter, and the sponsor always had complete control as to what was on the top of the priority list. Over a period of years, it became simply a balanced flow of useful software in one direction and money in the other direction.

If the sponsor trusts the contracted company, this is the simplest contract to use. The difficulty is that it requires a certain level of trust. What some contractors do is to request a trial period. Using the agile approach, they deliver usable functionality early and so establish the trust of the sponsor for subsequent work (see the house construction example on page 327).

Not-to-Exceed with Fixed-Fee (NTE/FF)

This is the contract method used in the airport construction example on page 329. It presupposed stable requirements. "Fixed fee" means that the contracted company is guaranteed a certain profit margin over their materials and subcontractors' fees. This protects the contracted company in case requirements are reduced or the work goes faster than estimated. "Not to exceed" puts a ceiling on the total amount paid to the contracted company, which protects the sponsor in case the work goes slower than expected. With protection for both sides, both the sponsors and the contracted company can look for ways to speed the work and live with unfortunate events. The airport construction project on page 329 used this form with Payment on incremental acceptance.

Fixed Price per Function Point or Story Point

When the customer and contracted company can agree on a unit of delivery, such as function points or story points, then they can settle on a price for each unit delivered. A number of contract houses these days estimate the size of a project in function points, create a price per function point delivered, and then have a certified function point auditor assess the actual number of function points delivered in the end. The customer then pays that amount, not the originally estimated amount. This is a nice contract form because it encourages the contracted company to work more efficiently (to increase their pay per delivered function point) and because it allows the customer to change the requirements along the way.

The same can be done with XP-style story points (Cohn 2005), except that there aren't any certified story point examiners to judge the final result. Bob Martin of Object Mentor posted an interesting variant to get around this problem: a base fee per story point, plus a lower-than-usual (close-to or below cost) fee per hour. This biases the contracted company to deliver early but gives them some protection in case work proceeds slower than expected. Martin described it this way:

"[A]gree to pay a certain amount for each point completed, plus a certain amount for each hour worked. For example, let's say you've got a project of 1,000 points. Let's also say that a team of four has established an estimated velocity of 50 points per week. This looks like about an 80 man-week job. At $100/hour this would be a $320,000 job. So let's reduce the hourly rate to $30/hour, and ask the customer for $224 per point.

"This sets up a very interesting dynamic. If the job really does take 80 man-weeks, it will cost the same. If it takes 100 man-weeks, it will cost $344,000. If it takes 70 man-weeks, it will cost $308,000. Notice that this is a small difference for a significant amount of time. Notice also that you, as developer, feel strong motivation to be done early, since that increases your true hourly rate." (Martin 2004)

I have not seen that model in action myself, but several people have written in recommending it.

Venture-Capital Financing Model

This can be used with any of the previous contract forms. In this model, the sponsor gives a round of financing for a certain amount of work, and the contracted company must produce results in order to get more funding. The sponsor can cut their losses at any time if they are not getting the results they need. They can presumably alter the terms of the contract after each work period. The result of a work period need not be working software; it could be a paper study, or a requirements document, or anything the sponsor selects.

The venture-capital finance model works well with agile providers because the agile provider is used to delivering useful, working software early and regularly.

I find it an odd irony that the venture capital financiers running start-ups that I have encountered don't take advantage of their own model to the extent that agile teams do. The venture financiers let the evaluation markers occur too far apart in time. If they attached funding to monthly releases, that would oblige the start-up team to think through what it really can accomplish each month. The monthly progress would give the financiers a better sense of the start-up company's real progress.

Incremental Delivery with Payment on Incremental Acceptance

This can be used with either fixed-everything contracts or the NTE/FF contract. The sponsor expects incremental delivery of integrated, tested systems, constructs and runs acceptance tests at each delivery increment, and pays the contracted company after each successful incremental acceptance. This was used in the airport construction project (p. 329) and in Solystic's French Post Office contract (p. 364). Both project leaders told me that this payment schedule has a wonderful motivating effect on the contracted company. The requirement, of course, is that the sponsors have an understanding of incremental development and can make acceptance tests for the incremental deliveries.

The next shaded box is an interview about taking agile into the customer space, with Eric Olafsson, the CEO of Tomax.

A few years ago, Tomax worked to fixed-everything contracts. They were, of course, problematic because something was certain to change even within a simple, three-month contract. Eric used the agile model to change the company's relationship with its customers. They now write contracts so that the customer gets early results through incremental development and can cancel or change direction after each increment. In exchange, the customer promises in the contract to make its experts available to the development team and to actively steer the project's direction.

The contract converts the customer from "them" to "us," part of the larger development team. The customer gains early returns and greater steering control. Tomax reduces its pricing risk and gets better input from the customer-side stakeholders and users.

Taking Agile into the Customer Space

by Eric Olafson

Could you set some context for the readers about Tomax?

Tomax makes solutions for retail chains. We bring together formerly disparate aspects of their business into what we call the demand-driven retail continuum. It's our thoughtnot unique to usthat retailers should be integrated from a people and process perspective, from merchandise planning through execution in the store, including how they manage their workforce. As simple as that sounds, what most retailers do is buy and integrate dozens of different solutions, each an island of technology, and play the systems integration game in perpetuity.

So there's a technology aspect, providing a Web model to manage retail. But as we become better at all we do, we reach out more and more to the business and we talk about the power of people simply working together better.

Today, we have 180 staff at our headquarters in Salt Lake City and another 70 in Bangalore. We currently are pretty much focused on the North American market but clearly looking at the international opportunity.

Could you talk a bit about bringing agile out of development, into the client relationship process?

You can call it the undocumented feature of agile: In an agile business model, the client, when everything is working right, is an author of the decisions being made. Thus is the ownership chemistry where the customer is always pleased with, or at least accommodating of, the realities of the project (i.e., the way things are), especially when they are making the decisions.

In the old model, the software company obligates itself to the customer for specific deliverables either at a fixed price or in an open-ended arrangement (the latter being more happy). In that model, the dynamics are all about the software company delivering things to the customerjust like a grocery store, just like in a restaurant, just like any other customer/vendor relationship.

We are trained in this mode: I am the customer, you are the vendor, so please behave accordingly.

In the agile model, it is subtle but important that the customer is part of that decision process. I don't think enough is understood about this subtlety by practitioners of agile to appreciate it. It offers the potential to drive self-regulation of a project, where the customer and the software vendor become true partners.

How does this work with specific customers?

Our first efforts involved selling the agile method. That went quite well. Then as we entered the contract phase, we saw conflict between traditional engagement mechanisms (the agreement) compared to what would be required in an agile environment. Fundamentally, the agile approach invalidated our contract. Our working materials for execution with our customers were inappropriate.

We needed, at this point, to take the customer to a framework that said: Here is not a specific list of deliverablesrather, here is a list of desired outcomes, preferred end states, goals, and objectives.

We had to get the customer to agree that the only thing we know for sure is that what we'll have at the end will be different from what we think it is today, although the business goal will not change.

We created a list of objectives and specific feature requirements that, in fact, were identical to the old way. But we created an approach to reconciling things based on the business objectives, the "this is what we're trying to obtain."

We laid out the prioritieswe got the customer to agree that in reconciling ourselves to what work should be done and what work shouldn't be done with these as the principles would determine what gets selected.

It was actually easy to get the customer to agree that, under all circumstances, these "business" principles should drive IT outcomes even if it came at the expense of desired function requirements.

I'm guessing that this had to be done with an existing customer, with whom you already had a trusted relationship.

This was a new customer. After we got that agreement, it wasn't difficult to proceed.

Where it got difficult was when we drifted back into traditional project-management methodology which. . .surprise. . . isn't agile-oriented. Our mistakes to this point were places where we didn't adhere to our own philosophy. It was too easy to get back into the separation of customer and vendor, where the customer becomes very comfortable in either liking or not liking things that are presented to them.

Had we been more scrupulous about maintaining the customer ownership dimension, I believe that those inconsistencies we experienced would not have occurred.

How hard is it to get staff time from the customers?

At the outset, quite easy. As you might expect, more difficult once into the daily grind. However, it continues to be the ingredient that produces the ownership which produces the partnership. If you don't do it, it won't happen.

The people in the agile community don't yet appreciate the impact on development caused by these co-driving relationships.

This needs to get figured out further by companies like ourselves and by others.

You've been working on bringing agile to this company since at least 2003. What's it like trying to drive change from the CEO position?

The popular notion is that the developers embrace agile first, and then suffer the efforts of their ignorant management in not seeing the lightthat ultimately, where it's successful is where it's driven by the developer community.

In our case of Tomax, it was different. Other members of the leadership team and I were exposed to agile at a time when we had become wholly dissatisfied with the traditional approach. I saw nothing but inefficiency in our efforts to have our consultants work with their customers' consultants to arrive at poorly understood ideas about what was required, and then turn around and give these to our developers in isolation, and then further dilute that understanding.

The idea that it's good for a developer to work with the customer, that they should communicate regularly, that they should deliver code more frequently, that things changenone of these things are revolutionary.

However, the notions of pair programming, daily stand-up meetings, information radiators, furniture arrangements, the veneer around that commonsense, those things can come across alien and foreign to the developer.

In our case, we found it was difficult to get people excited about it. In all honesty, it may have been more of a culture thing, in that we had been too long, too far down the bad way, so change was difficult. We worked to get people exposed to the commonsense, and to try it and see the benefit.

It is documented adequately elsewhere that trial on a limited basis, successfully undertaken, will lead to a more pervasive change.

I can't imagine that it was quite that easy at the beginning. I know that you weren't alone in trying to bring this to the company, without success for at least the first year.

In our case, trial ultimately got us there. But not so easily.

At the same time that I was working from the top, a zealous change agent was working in the development ranks. In a sense, neither of us got very far. The change agent eventually leftI'm starting to think that it is one of the characteristics of change agents that they almost need to get burned up in the process. People sometimes become more open to the exact same change after the change agent has left.

Where was the inflection point, then?

It was not until we had a number of people exposed to the implications of agile development that we reached an inflection point.

In our case, "exposed" meant that more than a dozen of our developers attended the agile conference in Salt Lake City in 2004. When they came back from that conference, they requested a number of changesto rearrange the furniture, for examplethat I would have been happy to offer them a year earlier. But this time it was their request.

After that, we got to a pod-based office layout, paper on the walls, reflection workshops, information radiators, more and closer interaction with the customers, and so on. The usual agile toolkit.

And now?

I would say that subsequent to our adoption of agile, our success has been based more in making agile go away and enforcing the notions of better customer relations, better code, improved product quality, and building better products. There are parts of our company where the use of agile terminology is more prevalent than in others. But the thing we celebrate is the customer interaction.

Agile is a constant pursuit. For example, there's not a person on our development staff who has not extolled, enthusiastically, the benefits of customer interaction. In some cases, developers come back from a customer visit and say that it's the best experience that they have had in our company. But then the very next week the same developer resists another opportunity to interact with the customer. People are built a certain way, and even some good experiences aren't enough to get them work differently.

Where are you heading at this time?

I think that deep and detailed domain knowledgeextreme domain expertise, if you willis where it's at; it's the secret sauce that really makes the agile methodology work.

Domain knowledge is taken as a given in the agile community; you have to have it. But genuine leaps in productivity occur when the domain knowledge provider has really detailed knowledge, vision, and clarity of purpose, plus the desire and the ability to work with and communicate with developers to produce a solution.

The developers, in turn, need to have a very deep understanding of the domain space, a genuine interestnot to be corny, but to live the problem and live it in detail.

This is a very important point. It's all about the details.

People truncate conversations and software design prematurely. Interestingly, our best design sessions exist at a level of specificity far beyond the concept, and much more where the rubber meets the road. It would be an oversimplification to say it's at the user interface level. It's about "What job is getting done?" to use Clayton Christensen's term.

Deep and detailed domain knowledge, in both the customer and the developerthat sounds like a rare combination.

We have the benefit of working with an extreme domain expert, Bernie Brennan, renowned in retail circles for his expertise in the retail industry. What is perhaps not so well appreciated is his keen interest and his understanding of the deepest details that make up almost everything going on in the retail continuum. At Tomax, we are taking the start in this direction.

Introducing Change into an Organization

Introducing change into an organization is scary and difficult. Agile development is doubly scary because it says

• First, change the way you do things.

• Next, keep changing, forever.

This puts agile development fully within the topic of introducing change into organizations, a very difficult topic, indeed.

The noted family psychotherapist Virginia Satir studied, among other things, people's reactions to change. A key observation of hers was "if there's ever a question between comfort and familiarity, familiarity will almost always win out" (Satir 1999). That is, even if the familiar mode is inefficient and uncomfortable, people stay with it.

The Process Miniature

A useful corollary of that observation is that things are easier the second timethey are more familiar. This shows one possible way out of the unfamiliarity trap: the Process Miniature (Cockburn 2005a).

A process miniature is nothing more or less than going through the new procedure in as short a time as possible: a minute, 15 minutes, an hour, a day, or a week, depending on the process.

• We can work through a simple example of the planning game (Beck 2001) or the blitz planning technique (Cockburn 2005a) in 15 minutes.

• In the "Extreme Hour" game,^[44] people run through two iterations of XP in one hour. There is a one-hour version used in showing people how Scrum works. I created a theater piece of Crystal Orange Web (see p. 344), running through two iterations of delivering real code on their system platform in 90 minutes.^[45]

  ^[44] See http://c2.com/cgi/wiki?ExtremeHour.

  ^[45] See http://c2.com/cgi/wiki?ExtremeHourWithActualProgramming.

• In my use-case course, I run a one-hour process miniature of gathering use cases, from defining system scope to stakeholders' interests, naming all use cases, and writing one use case completely to show how all the parts of the process fit together.

• One company had every new employee go through a full development and deployment cycle in one week to learn their company's process. After having completed that initial mini-project, the new employee could be placed on any project, in any stage of development, and would have an understanding of what was going on.

Going through a process miniature creates in the mind what Grady Booch neatly but mysteriously called a gestalt round trip. After the gestalt round trip is completed, the process exists as an entirety in the mind, can be viewed internally, and importantly for us here, is a bit more familiar and less frightening.

Anchoring New Behaviors

The process miniature also creates what sociologist Karl Weick calls a 'small win' (Weick 1979). Weick found that small wins strengthen a group's strength and self-image. Good groups look for early, small wins to strengthen their sense of team accomplishment. One of these that system designers use is the Walking Skeleton architectural strategy ("a tiny implementation of the system that performs a small end-to-end function"Cockburn 2005a pp. 4950).

One of the surprises to me was that people get mileage out of small rewards, even very small rewards. Evidently, these also count as small wins in the mind of the receiver.

I was on the receiving end of this technique one time: A colleague visiting my house saw how my assistant had created a "project wall" showing all my ongoing projects and trips. He said, "Wow, that's great! Here, I'll give you a quarter for showing me such a great idea." A few minutes later he found something else that he liked and gave me another quarter. I found myself irresistibly drawn to offering him ideas and looking to see which he thought was worth his little token offering. I managed to get six of those quarters by the end of his visit, and I still keep them in a special place as a memory of his appreciation.

You might not think that a token as small as a sticker would have any impact on people's behavior, but just the act of giving and receiving a token of appreciation or accomplishment has meaning to the receiver. Here is Kay Johansen's story of using stickers to help create change.

Creating Change with Stickers

by Kay Johansen

Our company's product was a variety of Internet services not amenable to commercial automated testing tools, so we needed a programming-intensive approach to automating the tests. As Test Manager, I put together a team of five testers, most of whom had some coding background.

Frustratingly, we made no progress on automation. Every day, we came to work, intending to program a new automated test, but we could always see the backlog of material that needed testing right now, manually. Being good citizens, we performed those manual tests instead of starting programming.

I knew something had to be done, or we'd never get ahead of this mountain of manual work. Because we were running iterations with XP-like "stories," I created stories for a test framework. After a few iterations, we had hacked together a framework and about three tests. That was as far as it went for a while. The mental switch required to figure out how to write effective automated tests seemed too great. And the mountain of manual testing was always there.

My first strategy was posting a large, visible chart with the number of tests on it. The number was three.

I waited for it to increase. We made little progress during the next couple months. Interest seemed low. I created stories for writing tests, and the team would sometimes obligingly force a test out or, as often as not, fail to complete their automated test story because they spent extra time doing a really good job on their manual testing stories. They were looking out for the customers, and I couldn't criticize them for that. Still, something needed to be done.

One day while at the grocery store, I noticed a package of shiny stickers; I think they were colored stars. On a whim, I bought the package and took it to work. At the start of the next iteration, I announced that anyone who completed an automated test would get a sticker. That caught their interest. After they finished laughing, they signed up for the few automated test stories with alacrity. My impression was that they finally felt they were not being perceived as "selfish" if they signed up to write automated teststhey had permission to do what they wanted to do anyway. Even better, they were competing, playing the game!

Over the next few weeks, the chart of automated tests shot up, even though there were no "prizes" for "winning" and no concept of an end date or final counting. The team was energized and enjoyed a spirit of friendly competition and cooperation. I did nothing more than count tests and hand out stickers.

Some team members collected stickers on their monitor for all to see. Others pasted them on 3 by 5 cards so they could carry them to meetings to better show them off. The tester who had no coding experience got a lot of help . . . and ended up with the most stickers.

Before long, an online, automatically updated, full-color chart replaced my big hand-drawn chart. Test problems and framework problems got solved quickly.

Soon we had a robust framework that even included a test script recorder.

The team grew confident and set a goal for 550 tests by the end of the year, which they easily achieved.

I eventually stopped handing out stickers, but the energy and confidence continued. One year later the team had well over 1,000 automated tests.

A Large Change is Many Small Ones

If people are afraid to make small changes in their habits, they are more afraid of large changes.

Mike Collins, an expert at lean development, addressed this problem in an interesting way at the O.C. Tanner Manufacturing Company in Salt Lake City (see the next shaded box). In reading the shaded box, you may note that his method draws on Satir's idea of growing familiarity and Weick's idea of small wins.

Mike understood that if he showed up at their custom-awards manufacturing factory with a design for a tenfold improvement in their system, he would encounter resistance from employees and management. Instead, he created a new production area with a minor set of modifications, from which they got a noticeable but not revolutionary improvement. He posted the results, reflected with the people involved about possible improvements, and created a second trial production area that was both smaller and faster.

When I first visited O.C. Tanner, they were using their third new production area. The original and the first trial production areas were still in use. The new one used about a quarter of the floor space of the original and filled contracts in about a third the time of the original.

At the time of this writing, the entire facility has been converted to fourth-generation "mini-factories," as they now call them. Production lead times have been reduced from over 18 days to about 6 hours, with a threefold increase in efficiency. With the smaller floor space, production workers get Osmotic Communication. An information radiator at the entrance to their production area lets themand their managerssee what is being worked on as well as the productivity effects of the improved system. They expect to double their efficiency again during 2006.

What I found impressive was not so much the improvements the team had made (which Mike might have been able to accomplish from his prior experience and lean manufacturing theory) as his patience in working through a sequence of small changes and victories rather than going for one big win. I note in particular that

• Each process got good enough business results to pay for itself as it proceeded.

• Because the workers in the production area were invited to help set the direction of improvements, there was better buy-in from the workers.

• Both management and workers get used to having an evolving production process as a normal part of their working climate.

Mike and I both noticed that he would almost certainly have met too much resistance to succeed if he had proposed the third or fourth layout at the very beginning.

Notice in the business literature the frequent reports of a company achieving breakthrough results by making many small changes and reflecting on those changes. The latest I read was about Out-back Steakhouse soliciting ideas from employees, trying them out in selected restaurants, and then deciding which ones to franchise.

These ideas tend to come from within the organization rather than from an outsider. This is the "we are tuning it for us" principle, something I consider a critical success factor for the adoption of new processes.

There is one other factor that assists in installing change: a threat to the company's survival from an external source. This is covered in the next section ("Programmers Read the Harvard Business Review").

Lean Manufacturing at O.C. Tanner

by Mike Collins

The O.C. Tanner Company produces and ships about 9,000 employee recognition awards a day. Our work can best be described as "mass customization," in that we provide several million combinations of awards with an average order size of 1.3 awards per order (91% of the orders are for one award).

Five years ago, we started down the lean manufacturing track. Over those five years:

• The time from when an order is placed until we ship the first piece has dropped from 18 days to six hours.

• Quality has risen from a low of 80% to 99%.

• On-time delivery has risen from a low of 60% to 99%.

• Work-in-process (WIP) has dropped from almost 500,000 pieces to less than 6,000.

WIP used to be so large partly because we used large batch sizes between stations but also because we ran production quantities at 10% or more over the actual order quantity to compensate for the quality problems. The large WIP meant we had people working full-time to move, find, expedite, and otherwise manage the WIP pieces.

To measure our production improvement, we use a Quality-Efficiency-Delivery (QED) metric that averages improvements in quality, efficiency, and on-time delivery of the current year over the previous year. The improvement score was 34% in each of the last two years. This is a notable achievement given that current performance levels with both quality and on-time delivery are near 99%, with efficiency nearly three times better than it was five years ago.

Even though virtually every system, process, and practice has been rethought, the most critical aspect of the change has been a change in how the people think about their jobs and the company. Changing people and culture is much more difficult than changing systems, processes, and practices. In fact, to effectively change systems, processes, and practices requires that people and culture be changed first or at least in parallel.

Most of the employees had been with the company for many years and were well entrenched in the routines of their work. The company was reasonably profitable with good employee benefits, and few saw any reason for change.

Initially, production consisted of 28 departments with each department working more or less independently of the others. Within each department, employees generally had only a single or perhaps two job skills. Work was passed in batches from employee to employee and from department to department. Most employees focused on their single skill job, and few of the employees made any suggestions for change, and even fewer ever participated in any change effort.

To make the progress realized to date has required several critical changes:

• Teams had to be implemented in which the collective creative thought of all can be harnessed.

O.C. Tanner went from 28 production departments to one department of 11 teams. Each team is in and of itself a self-directed mini-factory capable of producing every award combination.

Working as a team member is not natural for those coming out of many of today's school systems. Think about it. School is largely about competition, about getting the best grade. It means that there must be both winners and losers. If, however, a team is to be successful, the focus must change to making every team member a winner. Working as a team member requires that we greatly improve many of our skills, such as our communication skills. This change in thinking is not an easy change.

• Management had to move out of the management mind-set and more into a mind-set of leadership.

The job of management is now much less of the day-to-day management of work and much more one of facilitating, teaching, and supporting the efforts of the teams.

The first change to management's thinking was to learn that teams not only can but most often will make better decisions than the manager. Most decision-making and creative improvement therefore had to move to the team members.

Because the manager often has had greater experience, he or she will see that the team on occasion is making a mistake. It is leadership that allows the mistake to be made, recognizing that we learn mostly from mistakes.

Sound judgment is, of course, necessary. The manager must balance the potential learning that comes from a mistake with the potential damage to the company. This change in thinking is also not an easy one.

• Team members had to learn multiple job skills and were expected to participate in many of the improvement efforts of the company. Most employees rotate between assignments during the day.

As the team members learn to do this, they find that they are more self-motivated, more involved, and more satisfied with their work experience. The jobs are less mundane.

Almost all of the employees will tell you that they would never go back to the old system.

• Training was a key factor.

Training meant not only job skill training but also training in many other areas, such as team skills, communication skills, creative problem solving, coaching, conflict resolution, etc. Training included hands-on efforts at making improvements and solving problems, and even the understanding and use of statistics, which is very often necessary to the making of good decisions.

At first one may even question the wisdom of making these changes. O.C. Tanner is learning, however, that it is only with time and only by doing that an organization can begin to see the benefits of staying on the lean track.

The change rate at O.C. Tanner is accelerating. We expect efficiency to double again in the next 12 months. Leadership, instead of continually pushing the company's employees for more, is now finding itself in a position of not being able to support or even stay on top of the amount of change now taking place. It is an enviable position.

Programmers Read the Harvard Business Review

An odd cultural shift has been the increasing number of programmers and testers who are reading the Harvard Business Review (HBR). This is a direct consequence of their watching the business consequences of their frequent deliveries and their prioritization choices. Enough technical leads have written to alert me to some article in the HBR that I ended up subscribing to it myself, just to keep up with what the agile development community was discussing.

As an example, the April 2006 issue (the time of this writing) contains two articles relevant to the topics in this book.

Ram Charan's article, "Home Depot's blueprint for Culture Change" (Charan 2006), details Home Depot's massive cultural change process as it went from a decentralized, entrepreneurial set of stores that could not or would not collaborate to a centrally orchestrated set of stores that did collaborate. The new CEO, Robert Nardelli, was brought in from GE exactly to make that sort of cultural change.

Nardelli had one force on his sidetheir competitor, Lowe's, was gaining quickly, and it soon became clear that Home Depot's survival was at stake. Nardelli used that force to make many changes very quickly. While hard, it had the advantage that results also showed up quickly.

When people complained about the pace of change, Nardelli was able to answer back, "Good point. Give me five minutes. I'm going to go call Lowe's and ask them to slow down for us."

This sort of external threat helps a lot in instigating change.

Key to their change process was the institution of metrics. I found it interesting to read that metrics were used to increase collaboration, the "we're all us" mind-set:

"At the same time, the metrics made clear and reinforced the collaborative behavior and attitudes that Nardelli and Donovan wanted to encourage. . . . Company-wide metrics also provided a platform for collaboration."

They used the metrics to show that they made a greater profit when working together than working separately or competing among themselves.

They instituted

• An annual strategic conference

• Learning forums for managers in which they ran simulations of various collaborative and competitive strategies

• Weekly conference calls among the top executives around the nation

Alert readers will recognize all of these as staples of the agile approach.

I noted two more key points in the article, the first concerning how long it takes to get a cultural change to stick, and the second about the importance of suggestions from inside the team:

"A year and a half after Nardelli took over as CEO, he and Donovan knew that there still was significant opposition within the organization to the changes they were making."

"Assuming the rate of change is more or less right, how do you make it stick? . . . Where possible, get people affected by a change to help define the problem and design the solution."

Note how similar these points are to those raised by Eric Olafson and Mike Collins in their shaded boxes.

The same issue of the HBR included an article about global localization (Rigby 2006):

"Standardized offerings discourage experimentation and are easy for competitors to copy . . . Customization encourages local experimentation and is difficult for competitors to track, let alone replicate. When well executed, location strategies can provide a durable competitive edge for retailers and product manufacturers alike."

". . . successful localization hinges on getting the balance right. Too much localization can corrupt the brand and lead to ballooning costs. Too much standardization can bring stagnation . . ."

The recommended strategy is to standardize within a related cluster of situations and localize separately in each cluster.

This strategy matches the discussion earlier of tailoring methodologies to projects (see "Methodologies across the organization," page 210).

Several organizations I have visited came to the conclusion that they could benefit from three base methodologies (for each major project type they encountered), and they should then tune the base methodology locally on the project. This allows them some scale advantages and some tailoring advantages.

There are two points I wish to draw from this development:

• Modern agile programmers do care about how their work influences the organization and are studying business management within their professional development as programmers.

• Agile development practices are applicable to business management life in general, and vice versa, as evidenced by professional business writings.

House Construction

Many people argue that house construction isn't (or is) like software development. Because I have been making extensions to my house for several years, I now have some first-hand evidence to offer.^[46]

^[46] Yes, I know this is not new house development. I know others who use the Walking Skeleton (Cockburn 2005a) and related agile strategies even with new house development.

You can probably imagine how I would go about making house extensions. They should be done incrementally, using time-and-materials (trust-based) plus venture-capital (cut your losses) funding in close collaboration with both the architect and construction contractors. You would be right.

In our first conversation, the architect suggested creating a "master plan" of all the desired changes and then doing the work incrementally. I declined because I was pretty sure that we would change too much for the initial plan to have meaning by the halfway point, and it would end up being a waste of my money. As events transpired, I was correct.

We chose to run each idea as a standalone project but were careful to ask at each key moment, "If we were to extend in [some particular way] next year, how would that affect our decision now?" Surprisingly, we found no cases where the future choice affected the present decision, including putting a partial basement under the house.

Our first project was to put a basement under our house.^[47] This turned out to be easier than expected because our house was built on short stilts to provide an insulating air cushion over the ground.

^[47] I am told that people usually put the basement in first. Putting it in last is an unintended but amusing echo of the XP strategy to plan a project as though features can be implemented in any order.

Rather than excavate the whole basement, we decided to excavate only a third of the basement. This gave us the basement entrance we needed and left open the question of whether we would extend the basement or build a side wing to the house (in the next project). We learned that there would be no cost savings for excavating more than we needed now, even if we chose to expand the basement later. The XP community calls this the YAGNI principle, for "You Aren't Gonna Need It." We excavated only what we needed. Three years later, we still have no plans to extend the floor space, either above or below ground. YAGNI is holding.

The architect and the contractor looked underneath the house to see how to support the house while digging underneath it. The architect launched into a dissertation on the various choices. The contractor cut in with, "Why don't we just run a giant beam right down the center and hold the whole thing up while we excavate?" Note here the application of the 11th principle of the agile manifesto:

"Simplicitythe art of maximizing the amount of work not doneis essential." (see p. 376)

The architect adopted the suggestion, and after that, the architect, the contractor, and the construction crew had excellent conversations that merged their best ideas. To this day they always trade construction ideas back and forth along with how those might change the design of the house itself.

A surprise hit us on the first day of construction. They knocked a hole in the concrete wall where they were going to excavate, peeked in, and discovered that the beams under that part of the house ran perpendicular to those where they had looked under the house. This ruined their plan for the support beams and showed that once again, civil engineering is ahead of software engineering. It usually takes software teams a week or two to invalidate plans. These people were able to do it within hours.

As we progressed, I quickly noticed that the contractor kept changing his "fixed-price" bid for the project as new information appeared. I finally suggested we drop the fixed-price façade and just work to time and materials. His reply surprised me: "If you are willing to carry the extra risk, that will be fine with us." I said, "I don't see any extra risk. You'll just change the price whenever something changes anyway." He said, "That's true, but most people don't recognize that." In exchange for my accepting the "extra risk," he lowered his profit margin from 13% to 10%. We each felt we had benefited from this change.

With the new contract in place, his crew was able to do any number of other small projects for us, such as changing the kitchen counters, adding closets, and fixing the fence around the yard. Neither these nor the other usual surprises, twists, and turns on the project worried either of us any more.

I noticed with some interest that he and his crew held a daily stand-up meeting^[48] each morning to set their day's work goals. In this short meeting, they ensured that they knew what they were working on and had the materials to get it done.

^[48] Part of the Scrum methodology (Schwaber 2002), also used in XP.

The project came to a successful conclusion, and we gave a small bonus to the contractor and the workers. Unbeknownst to me, our prompt payments made us preferred customers, which proved useful for the next projects.

The second project was to sculpt the yard, move in stone slabs, and do some general landscaping. Although this was a "minor" project, the contractor was delighted to help. He sent digging and moving crews to help when we needed it and even operated machinery himself when his key men were ill. This was part of us being preferred customers, a roll-forward benefit from the first project.

The third project was to add a two-story porch to the side of the house. At this point, we were glad not to have done a master plan because this addition was completely different from anything we originally had in mind.

By now, there was trust and good communication between the architect, the contractor, and us. The small wins with reflection, the venture capital funding model, and the willingness to dance with the situation were all paying off. The contractor passed along our preferred customer rating to his suppliers, so we got extremely fast service. He also took it upon himself to see that they gave us good rates, so we saved money.

Because we contracted by time and materials, it was natural to have the sub-contractors do additional projects around the house as we needed it. The final bill for all this extra work was much lower than it would have been had we contracted each one separately.

While the contractors are finishing the side porch, the architect is starting on a redesign of the front porch. His design is quite different from what he originally proposed when he was enamored of his master plan. That is because we now have all the other major changes to the house in place, and he can see how to tie them together in the front approach.

The point of this long story is to illustrate how the lessons from agile software development transfer to a completely different field. Incremental development, architectural changes, daily standups, YAGNI, time-and-material contracts, venture capital funding, customer involvement and steering, small wins, process miniature, and developing collaboration and trusteach of these proved useful.

Airport Construction

There might be those of you who think that house construction is too easy. I was delighted to meet an architect who uses the agile processes in airport design and construction.

On a flight to Boston, I sat next to Joe Wolfe, who answered in reply to my question about his occupation that he designs airport terminals (Terminal E in Salt Lake City and the new Delta terminal at Logan Airport). Not being able to resist, I asked how he worked. His reply astonished me because he recounted almost point for point the Crystal Clear methodologyexcept he was doing it with several dozen subcontractors on airport terminal construction! I showed him my draft of Crystal Clear, including the blitz planning technique, and he said, "Oh yes, that's what we do."

At the beginning of the project, he collects the subcontractors in a room and has them brainstorm the work they will have to do. They write their tasks and time estimates on sticky notes and post them on a large wall. They sequence the tasks on the wall until they have a sensible initial plan that shows both tasks and dependencies. Someone transcribes the contents of the notes into a computer after the session.

• I asked Joe whether he had a notion of early integration in his project and how he would motivate subcontractors to do that. He said they did do that; the subcontractors get paid at integration milestones, and the cash flow motivates them.

• In reply to my question about contracts, he introduced me to "not-to-exceed plus fixed-fee" contracts (see "Contracts," page 312). "Not to exceed" means that the subcontractor promises not to exceed a certain final amount. "Fixed fee" means that the subcontractor is guaranteed a certain profit if the work takes less time than expected. Thus, the subcontractor's profit is protected, and the airport's spending bill is protected.

• I asked him about the agile notion of exposing bad news earlyhow would he get subcontractors to reveal their problems early? He said that incremental funding motivates them to integrate their work and also to seek help when they run into trouble. An example might be an international shortage of steel.

• I was concerned that even with those incentives, subcontractors are not used to exposing their problems to the people hiring them, and it must take some time to get them used to it. He highlighted that he, as project coordinator, had to deliberately build a climate of trust within the team so that they would reveal their problems.

Note the similarity of his working style with that described in the discussion of personal safety in Crystal Clear (Cockburn 2005a, p. 29):

"Skillful leaders expose their team members (and themselves!) to these situations early, and then demonstrate with speed and authenticity that not only will damage not accrue, but that the leader and the team as a whole will act to support the person.

"One project leader^[49] told me that when a new person joined her team, she would visit that person privately to discuss his work and progress, and wait for the inevitable moment when he had to admit he hadn't done or didn't know something.

"This was the crucial moment to her, because until he revealed a weakness, she couldn't demonstrate to him that she would cover for him or get him assistance. She knew she was not going to get both reliable information and full cooperation from him until he understood properly that when he revealed a weakness or mistake, he would actually get assistance. She said that some people got the message after her first visit, while others needed several demonstrations before opening up."

^[49] Thanks to Victoria Einarsson in Sweden.

I was stunned by Joe Wolfe's account of designing and building an airport, as it didn't match my preconceptions at all. As a final question, I asked him if this way of working was normal in the airport-designing industry. He said, no, it wasn't, but he couldn't see how anyone could work in any other way and get the terminal completed on time. He added that he had been working this way for several decades, and his clients were so happy that he never had to look for work.

Book Publication

Finally, one can apply agile development to book publication. I did this for the first edition of this book. The normal book publishing process takes about four months. Because we had only two months before the conference announcing the book, I asked to manage the publication process myself, using the principles in this book.

The first thing I did was to select people all living in Salt Lake Citythe closest I could get to full collocation. In a normal production process at that time, the manuscript would be mailed in paper form from person to person, with pencil marks on it for changes to be made. These days, the manuscript is marked electronically and emailed, but that only partially improves the matter.

Having every specialty located in Salt Lake City is not quite Osmotic Communication, but at least we could all meet at someone's house to discuss the manuscript. Naturally, we used electronic markup and email instead of paper.

The second change was to work incrementally. Most of the editors I have encountered like to mark up the entire manuscript in one pass. This is largely a consequence of people working separately across large distances. Knowing we would meet periodically, we arranged to build each chapter separately. I made a dependency grid marking the work needing to be done by each person so that the illustrator could work at a different pace and on different chapters than the copy-editor and the page layout person. I allocated a certain number of rework cycles for each person. This way, each person could be busy doing their work as independently as possible from each other person.

I met with the copyeditor after she marked up each of the first several chapters so that we could synchronize the style and nature of the changes. As we formed agreement on what counted as a mistake versus what to leave as author's style, the number of marks she had to make got smaller, and the number of corrections I had to make to her corrections got smaller. This was an improvement over the usual mode, in which the copy-editor marks up the entire manuscript, and then the author has to unmark any parts that are style instead of mistake.

On the final day, we met at the house of the page layout person. Several times we found ourselves working around an awkward column break and correcting sentence structure at the same time. With all of us sitting together, sometimes I would revise a sentence to make it fit before a column break, and sometimes the layout person would reduce the interline spacing by a small amount to make the extra line fit within the vertical space.

This experience with book production taught me several things.

• Agile development practices work very well with book production. We reduced production time from four months to three weeks.

• The copy editor was never comfortable with the process we used. One reason was that she never knew what state the book was in at any point in time (she was not accustomed to the high flux). However, I came to realize that one of her pleasures in life was sitting down with a paper manuscript and a pot of tea, and having a quiet morning marking up the text with her pencil. Working online, working one chapter at a time, and running out to meetings with the author did not suit her working style. (Does this sound like any programmers you know?)

• There was an extra cost to working the way we did. When she submitted her invoice, she wrote that the invoice was higher than she originally bid because the author made last-minute changes that cost her time, and also because of the extra meetings she had to attend. (Does this sound like any programmers you know?) This matches the prediction of the economic model in Chapter 4 (p. 164). That model shows that concurrent development should be faster but more expensive than sequential development. We worked about four times faster at about a 10% cost increase.

• Finally, not everyone enjoys iterative and concurrent development. At one point, when the copy editor was resisting meeting me to discuss the mark-up, I started explaining the benefits of our get-together. She cut me off, saying, "I knowI'm editing that chapter!" She was happy afterwards to go back to her paper manuscripts and pot of tea and put agile book production in her past (for the time being!).

Conference Organization and Limits of the Agile Model

I can think of only two environmental factors that limit the applicability of the agile model: distributed teams and inability to use incremental development.

Distributing the team slows communication and feedback, which makes it more difficult, though not impossible, to use the agile approach. This was discussed at length in "Agile teams must be colocated" on page 246. When working in a distributed team situation, more care must be placed on following the agile principles in general.

Any situation in which incremental development can't be done loses some of the benefits of the agile approach. Space exploration is one.

The incremental version of putting a man on the moon would be to put his foot on the moon in the first space shot, and then to add other parts of his body and equipment on subsequent space shots. Clearly that doesn't make sense. What we employ instead is the use of iterated prototypes: putting a whole man in orbit, putting a man and a craft around the moon, and then, finally, ten years later, putting a whole man on the moon.

Note that all the other agile principles can be used even in space exploration.

Possibly the most difficult situation to try to apply the agile model is in organizing a conference, particularly with volunteers (as is often the case) located in different parts of the world (as is often the case). There are no iterations, there are no increments, there is no opportunity for feedback, there are no users to get feedback from.

Here is our story from organizing the Agile Development Conference of 2003 and 2004 and how we "ate our own agile dog food."

• Before we pitched the conference to the first potential hosting group in 2001, we created a vision statement for the conference that stated why a new conference should come into existence and how each part of it fit into the overall purpose (Cockburn 2003a). Clear vision statements are part of aligning teams. They are core to agile projects because they allow people to move fast, in harmony, and with less bureaucracy than otherwise.

• We recognized that while incremental and iterative development aren't meaningful to the organization of a single conference, they are meaningful to the organization of an annual conference. We therefore ran numerous reflection and feedback sessions, starting on the last day of the 2003 conference. We collected from both attendees and organizers what they would like to keep the same or change for the next year.

• We compared the feedback comments to the original vision and decided whether it was our vision or just our implementation of the vision that needed correction.

• Some of the people organizing the following year's conference sat around my house with our feet up (and some with beer in hand), discussing the differences between software development and conference organization. We wanted to see what might be done to copy into conference organization what we knew worked in software development (notice our use of a face-to-face setting and a relaxed atmosphere to spark invention).

• We decided to collocate most of the conference organization committee. We established the idea of a "design epicenter" for the conference. The design epicenter could be in a different place than the conference. That would allow us to choose a geographically constrained region where we could find all the needed track chairs. We selected London as the design epicenter for 2004 and Silicon Valley as the one for 2005.

• The London-based track chairs met periodically at pubs to discuss ways to better integrate the different conference tracks. I attended a few of these, and even I was surprised at the number of new ideas that showed up at each meeting. The group made more progress in a single evening together than in a month of emails. (The ideas were surprises; that they made such progress was notit was why they met at pubs in the first place.)

• One of the better innovations was to borrow from the apprenticeship model. We decided that the following year's track chairs would work as understudies to the London group. Then, when it was their turn to act as track chairs, they would come to the topic warmed up, understanding the thoughts that had gone into the various decisions (this is borrowing from Peter Naur's theory building model; see Appendix A).

• A lesson from the first Agile Development Conference (ADC) was that even the Open Space sessions (Owen 1997) were too formal for what we intended. People still enjoyed the conversations in the hallway the most. We therefore created the position of Hallway Chair. This person was to see that the hallway itself could maximize communication and community. The things we decided as crucial in the Hallway Session were:

  • A hallway (don't move it into a room and give it a session name)

  • Coffee (make sure the hotel staff never clear away the refreshments, but make sure they are available and fresh all day)

  • Tables where programmers could sit and show each other their special tricks and interests

  • Lots of butcher paper and whiteboards for people to draw on while standing around and talking

The ADC somewhat typically became the victim of a hostile takeover and was merged with a competing conference. Many innovations were lost: the idea of a design epicenter, the apprenticeship model, the use of a vision statement, the reflection workshops following the conference. As predicted in the cooperative game model, with the change of team came a loss in the understanding of what made the conference special.^[50]

^[50] The continued presence of Todd Little kept most of the conference structure together. Todd helped design the first ADC, chaired the second, and then chaired the first two years of the merged conference. If Todd had not continued, much more would have been lost.

While I feel sad about the loss of ADC's innovative ideas and vision, all the previous is to be expected in business, whether agile or otherwise. It is partially satisfying to see that the theory predicts properly.

I catalog these ideas here in case someone else has occasion to use them in their own conference design.