Becoming Self-Adapting

If you have been reading this book from the beginning, you should still see one mystery at this point.

Every person is different, every project is different, and each project differs internally across subject areas, subsystems, subteams, and time. Each situation calls for a different methodology (set of group conventions).

The mystery is how to construct a different methodology for each situation, without spending so much time designing the methodology that the team doesn't deliver software. You also don't want everyone on your project to have to become a methodology expert.

I hope you can guess what's coming.

Bother to Reflect

The trick to fitting your conventions to your ever-changing needs is to do two things, individually and as a team:

1. Bother to think about what you are doing.

2. Have the team spend one hour together every other week reflecting on its working habits.

If you do these two things, you can make your methodology effective, agile, and tailored to your situation. If you can't do that, well . . . you will stay where you are.

Although the magical ingredient is remarkably simple, it is quite difficult to pull off, given people's funky nature. People usually resist having the meeting. In some organizations, the distrust level is so high that people will not speak at such a get-together.

In this case, there is only one thing to do:

Do it once, post the results, and then see if you can do it again.

You may need to find someone within your organization who has the personal skills to make the meeting work. You may need to go outside your organization for the first few times, to get someone with the right personal skills and whom everyone in the room can accept.

A Methodology-Growing Technique

Here is a technique for on-the-fly methodology construction and tuning. I present it as what to do at five different times:

• Right now

• At the start of the project

• In the middle of the first increment

• After each increment

• In the middle of subsequent increments

After that, I describe a sample one-hour reflection workshop.

Right Now

Discover the strengths and weaknesses of your organization through short project interviews.

You can do this at the start of the project, but you can also do this right away, regardless of where you are in any project. The information will do you good in all cases, and you can start to build your own project interview collection.

Ideally, have several people interview several other people each, and start your collection with six to ten interview reports. It is useful but not critical to interview more than one person on one project. For example, you might talk to any two of the following: the project manager, the team lead, a user interface designer, and a programmer. Their different perspectives on the same project will prove informative. Even more informative, however, will be the common responses across multiple projects.

The important thing to keep in mind is that whatever the interviewee says is relevant. During an interview, I don't give my own opinions about any matter but use my judgement to select a next question to ask.

I suggest that you structure your interviews in this sequence as well:

1. Ask to see one sample of each work product produced.

2. Ask for a short history of the project.

3. Ask what should be changed next time.

4. Ask what should be repeated next time.

5. Identify priorities.

6. Find any holes.

Step 1. Ask to see one sample of each work product produced

Looking at these, you can detect how much bureaucracy was likely to be on the project and see what questions you should ask about the work products.

Look for duplicated work, places where the work products might have been difficult to keep up to date.

Ask whether iterative development was in use, and if so, how the documents were updated in following iterations.

Look, in particular, for ways in which informal communication was used to patch over inconsistencies in the paperwork.

Work Product Redundancy

On one project, the team lead showed me 23 work products.

I noticed a fair degree of overlap among them, and so I asked if the later ones were generated by tools from the earlier ones.

The team lead said no, the people had to reenter them from scratch.

So I followed up by asking how the people felt about this. He said they really hated it, but he made them do it anyway.

After looking at the work samples,

Step 2. Ask for a short history of the project

Record the date started, staff changes (growing and shrinking), team structure, the emotionally high and low points of the project life.

Do this to calibrate the size and type of the project and to detect where there may be other interesting questions to ask.

Discovering Incremental Development

That is how I learned the fascinating story about the project I call "Ingrid" (Cockburn 1998).

During just the project inception phase, the team had hit most of the failure indicators I knew at the time. That their first four-month increment was a catastrophe came as no surprise to me. I even wondered why I had traveled so far just to hear about such an obvious failure.

The surprise was in what they did after that.

After that first increment, they changed almost everything about the project. I had never seen that done before.

Four months later, they rebuilt the project againnot as drastically, but enough to make a difference.

Every four months they delivered running, tested software, and then they sat down to examine what they were doing and how to get better (just as I am asking you to do).

The most amazing thing was that they didn't just talk about changing their way of working, they actually changed their way of working.

The value of this interview lay not in our discussing deliverables but in my hearing their phenomenal determination to succeed and their willingness to changeevery four monthswhatever was necessary to get the success they wanted.

After hearing the history of the project and listening for interesting threads of inquiry to pursue,

Step 3. Ask what should be changed next time

Ask, "What were the key things you did wrong that you wouldn't want to repeat on your next project?"

Write down whatever they say, and fish around for related issues to investigate.

After hearing the things not to repeat,

Step 4. Ask what should be repeated the next time

Ask, "What were the key things you did right that you would certainly like to preserve in your next project?"

Write down whatever they say. If the person says, "Well, the Thursday afternoon volleyball games were really good," write that down.

Getting Seriously Drunk Together

Once when I asked this question (in Scandinavia), a person said, "Getting seriously drunk together."

We went out and practiced that night, and I did, indeed, see improved teamwork between the people the next day.

In response to this question, people have named everything from where they sit, to having food in the refrigerator, to social activities, communication channels, software tools, software architecture, and domain modeling. Whatever you hear, write it down.

Step 5. Identify priorities

Ask, "What are your priorities with respect to the things you liked on the project? What is most critical to keep, and what is most negotiable?" Write those down.

It is useful to ask at this point, "Was there anything that surprised you about the project?"

Step 6. Find any holes

Finally, ask whether there is anything else you should hear about, and see where that goes.

At one company, we constructed a two-page interview template on which to write the results, so we could exchange them easily. That template contained the following sections:

1. Project name, job of person interviewed (the interviewee remains anonymous)

2. Project data (start/end dates, maximum staff, target domain, technology in use)

3. Project history

4. Did wrong/would not repeat

5. Did right/would preserve

6. Priorities

7. Other

Do this exercise, collect the filled-in templates, and look them over. Depending on your situation, you might have each interviewer talk about the interview, or you may just all read the notes.

Look for common themes across the projects.

The Communication Theme

At the company where we created the template, one theme showed up across the projects:

"When we have good communications with the customer sponsors and within the team, we have a good outcome. When we don't have good communications, we don't have good results."

Although that may seem trivially true, it seldom gets written down and attended to. In fact, within a year of that result, the following story occurred at that company:

The Communication Theme in Action

Mine was one of three projects going on at the same time, each of which involved small teams with people sitting in several cities.

As you would expect I spent a great deal of energy on communications with the sponsors and programmers.

The three projects were completed at about the same time. The director of development asked me what the difference could bewhy the project I was on was successful while the other two that ran at the same time were unsuccessful.

Recalling the project interviews, I suggested it might have something to do with the quality of communication between the development and sponsoring groups, and within the team.

He said this was an interesting idea. Both the programmers and the sponsors on the other projects had reported problems in communicating with their project leads. Both programmers and sponsors had felt isolated. The sponsors of my project, on the other hand, had been very happy with the communications.

The theme was different in another company. Here is what one interviewee told me:

The Cultural Gap Theme

Our user interface designers all have Ph.D.s in psychology and sit together several floors above the programmers.

There is an educational, a cultural, and a physical gap between them and the programmers.

We have some difficulty due to the different approach of these people, and to the distance we sit from them.

This company will need extra mechanisms to increase contact between those two groups of people and extra reviews of their work.

The point of these stories is to highlight that what you learn in the interviews is likely to be relevant on your next project. Pay attention to the warnings that come up in the project interviews.

At the Start of the Project

Expect to do some tailoring to the corporate methodology standard. This will be needed whether the base methodology is ISO9001, XP, RUP, Crystal, or a local brew.

Stage 1: Base methodology to be tuned

If possible, have two people work together on creating the base methodology proposal for the project. It will go faster, they will spot each other's errors, and they will help each other come up with ideas.

They have four steps to go through:

1.	Determine how many people are going to be coordinated and identify their geographic distribution (see the grid in Figure 4-23). Decide what level of correctness is expected of this software and what degree of damage it could cause. Determine and write down the priorities for the project: time to market, correctness, or whatever they may be.
2.	Using the methodology design principles from Chapter 4, select the basic parameters for the methodology: how tight the standards need to be, the extent of documentation needed, the ceremony in the reviews, the increment length (the time period until running code is delivered to real, even if sample, users).

If the increment length is longer than four months, they will have to find some way to create a tested, running version of the system every four months or less, to simulate having real increments.

3.

Select a base for the methodology, one that is not too different from the way in which they would like to work.

Recall that it is easier to modify an existing methodology than to invent one from scratch. They may choose to start from the corporate standard, the published Unified Process, XP, Crystal Clear, Crystal Orange, or the last project's methodology.

4.

Boil the methodology down to the basic work flow involvedwho hands what to whomand the conventions they think the group should agree to.

These steps could take between a day and a few days to complete for a small- or medium-sized project. If it looks like they will spend more than a week on it, then get one or two more people from the project team involved and drive it to completion in just two more days.

Stage 2: The starter methodology

Hold a team meeting to discuss the base methodology's work flow and conventions, and adjust it to become the starter methodology. For larger projects, where it is impractical to gather the whole team, gather the key representatives of each job role.

The purpose of the meeting is to

• Catch embellishments

• Look for ways to streamline the process and ways to communicate with less cost

• Detect other issues that were not spotted in the base methodology draft

Consider these questions in that meeting:

• How long are the iterations and increments to be (and what is the difference)?

• Where will people sit?

• What can be done to keep communication and morale high?

• Which work products and reviews will be needed, at what ceremony levels?

• Which standards for tools, drawings, tests, and code are mandatory, and which are just recommended?

• How will time reporting be done?

• Which other conventions should be set initially, and which might be evolved over time?

An important agenda item for the meeting is selecting a way for the team to detect morale and communication problems.

The meeting results will include:

• Basic work flow

• Hand-off criteria between roles, particularly including overlapped development and declaration milestones

• Draft standards or conventions to be followed

• Peculiarities of communication to be practiced

This is your starter methodology.

The meeting could take half a day but should not exceed one day.

In the Middle of the First Increment

Whether your increment length is two weeks or three months, run a small interview with the team members, individually or in a group meeting, at approximately the mid-point of the increment. Allow one to three hours.

The single question for resolution is,

"Are we going to make it, working the way we are working?"

In the first increment, you can't afford to change your group's whole way of working unless it is catastrophically broken. What you are looking for is to get safely to your first delivery. If the starter methodology will hold up that long, you will have more time, more insight, and a better moment to adjust it, after you have successfully made your first delivery.

Therefore, the purpose of this interview or meeting is to detect whether something is critically wrong and whether the first delivery will fail.

If you discover that the team's way of working isn't working, first consider reducing the scope of the first delivery.

Most teams overstate how much they can deliver in the first increment. This is simply normal and not a fault of the methodology. It is a result of overambitious management driving the schedule unrealistically and overly optimistic developers who overlook the learning to be done, the meetings to be held, and the normal bugs they put into the code. It comes from underestimating the learning curve of new technology and new teammates. Overstating how much can be delivered in the first increment is really quite normal.

Therefore, your first approach is to reduce scope.

You may, however, discover that reducing scope will not be sufficient. You may discover that the requirements are incomprehensible to the programmers or that the architects won't get their architecture specification finished in time.

If this is the case, then you need to react quickly and find a new way of working. This, combined with a drastically reduced functional scope, will allow you to meet that first delivery deadline.

You may introduce overlapped development or put people physically closer together, cut down the ambition level for the initial architecture or make greater use of informal communication channels. You may have to make emergency staff changes or introduce emergency training, consulting, or experienced contractors.

Your goal is to deliver something: some small, running, tested code in the first increment. This is a critical success factor on a project (Cockburn 1998). After you deliver this first release, you will have time to pause and consider what is happening.

After Each Increment

Hold a team reflection workshop after each increment.

Bothering to reflect is a critical success factor in evolving a successful methodology, just as incremental development is a critical success factor in delivering software.

The length of this reflection workshop may vary from company to company or country to country. Americans like to be always busy, short of money, and on the run. I see Americans allocating only two to four hours for this workshop. In other parts of the world, the workshop may be given more time.

I once participated in a two-day offsite version that combined reflection, team building, and planning for the next increment. It took place in Europe, not surprisingly.

The dominant reason for delaying this workshop until after the first increment is that you can only properly evaluate the effects of each element in your methodology after you have delivered running, tested software to a user. Only then can you see what was overdone and what was underdone.

There is a second reason for holding the workshop at the end of the increment: People are quite often exhausted after getting the software out the door. This meeting provides a chance to breathe and reflect. Done regularly, it becomes part of the project rhythm. After each increment, the team members benefit from a short shifting of mental and social gears.

Whether you take two hours or two days, the two questions you want to address are:

1. "What did we learn?"

2. "What can we do better?"

The responses may cross every boundary of the project, from management intervention to timecards, group communication, seating, project reviews, standards, and team composition.

Very often, teams tighten standards after the first increment, get more training, streamline the work flow, increase testing, and reorganize the teaming structures.

The changes will be much smaller after the second and subsequent increments, because by then the team will have already delivered several times.

In the Middle of the Subsequent Increments

After the first increment, the team has established one (barely) successful way of working. This is a methodology design to fall back on, if needed.

Having that as a fallback plan, you can be much more adventuresome in suggesting changes in the mid-increment meetings you hold in the second and later increments.

In those mid-increment meetings, and particularly after the second successful delivery, look to invent new and better ways of delivering.

See if you can do any of the following:

• Cut out entire sections of the methodology.

• Do more concurrent development.

• Use informal communications more to bind the project information.

• Introduce new and better testing frameworks.

• Introduce new and better test-writing habits.

• Get closer collaboration between the key groups in the project: between domain and usage experts, programmers, testers, training people, the customer care center, and the people doing field repair.

You might use interviews or a reflection workshop for these mid-increment adjustments. By this time, your team will have had quite a bit of practice with these meetings and will have an idea of how to behave.

You can omit the mid-increment workshops if the project is using increments of three weeks or less.

Why bother with mid-increment reviews, when the project is already delivering and you already have post-increment reviews in place?

In the middle of the development cycle, those things that are not working properly are right in people's faces. The details of the problems will not be as clear four or six weeks later, at the post-increment meeting. Therefore, you can pick up more details in the middle of the increment, get feedback immediately about the idea, and try out a new idea the same day instead of waiting several weeks or months.

What if a new idea doesn't work out? Sometimes the team tries a new idea on the second or third increment and finds that the idea simply does not work well.

Mid-Project Team Structure Changes

On one project, we went through three different team structures during the third increment.

A short way into the third increment, we decided that the team structure we had been using was weak. So we chose a new team structure to use on increment three.

It was catastrophically bad. We knew within two weeks that we had to change it immediately.

Rather than revert to the original, awkward-but-successful team structure, we created a new suggestion and tried it out right away.

It turned out to be successful, and we kept it for the duration of the project.

In inventing new ways of working in these later increments, you create the opportunity to significantly improve your methodology. This is an opportunity not to be missed.

The Post-Project Review

Given the mid- and post-increment reflection workshops, I place less emphasis on having post-project reviews. I feel that the time to reflect is during the project, when the reflection and discussion will do the project some good. After the project, it is too late.

Usually, I find that teams that run post-project reviews did not bother to reflect during the project and suddenly want to know how to do better for the next project. If you find yourself in such a meeting, put forward the suggestion that next time you want to use incremental development and hold post-increment reviews instead.

Nonetheless, it may be that the post-project review is the only time you get to make statements regarding the staffing and project management used. If this is the case, I suggest getting and using the book Project Retrospectives (Kerth 2001), which describes how to run a two-day post-project review.

If you hold a post-project review, think about who is going to make use of the information and what they can really use as they run their next project. You might draft a short (two-page) set of notes for the next project team to read, outlining the lessons learned from this project.

Of course, you might write yourself a one-page "lessons learned" reminder after each of your own increments, as a normal outcome of your reflection workshop.

A Reflection Workshop Technique

The tangible output of a mid- or post-increment reflection workshop is a flip-chart that is then posted on the wall in some prominently visible place and seen by the project participants as they go about their business.

I like to write directly on the flipchart that will be posted. It is the one that contains the group memories. Other people like to copy the list from the scratched- and scribbled-on flipchart to a fresh sheet for posting. The people who created the one shown in Figure 3-10 decided to use sticky notes instead of writing on the flipchart.

A Sample Reflection Workshop Technique

There are several different formats for running the workshop and for sharing the results (of course). I tend to run the simplest version I can think of. It goes approximately like this (see Cockburn 2005 and Tabaka 2006 for more detailed versions):

A Reflection Workshop

Hi. Welcome to this workshop to reflect on how we can get better at producing our software.

The purpose of this meeting is not to point fingers, to blame people, or to escape blame. It is to announce where we are getting stuck and nominate ideas for getting past that stuckness.

The outcome of this workshop will be a single flipchart on which we'll write the ideas we intend to try out during the next increment, the things we want to keep in mind as we work.

Let's break this flipchart into three pieces.

On the left side, let's capture the things we are doing well that we want to make sure we don't lose in the next increment.

On the right side, let's capture the new things we want to focus on doing.

On the supposition that the list of what we're doing right will be the shorter of the two, let's write down the major problems we're fighting with, halfway down the left side here (see Figure).

Let's start with what we're doing right. Is there anything that we're doing right, that we want to make sure we keep around for the next increment?

At this point some discussion ensues. It is possible that someone starts naming problem areas, instead of strong areas. If the problems are significant, write them down under the Problems section. Allow some time for people to reflect and discuss.

Figure 5.1. Sample poster from reflection workshop.

Eventually, move the discussion along:

All right. What are some of the key problems we had the last time, and what can we do to improve things?

Write as little as possible in the Problems section: Write as few words as possible, and merge problems together if possible. The point of this poster is to post suggestions for improvement, not to focus on problems.

Collect the suggestions. If the list gets very long, question how many new practices or habits the group really wants to take on during this next period. It is sometimes depressing to see an enormous list of reminders. It is more effective to have a shorter list of things to focus on. Writing on a single flipchart with a fat flipchart pen is a nice, self-limiting way of handling this.

Periodically, see if someone has thought of more things the team is doing right that should be kept.

Toward the end of the workshop, review the list. See if people really agree with the new ideas or if they were just being quiet.

After the workshop, post the list where everyone can see it.

At the start of the next workshop, you might bring in the poster from the previous workshop and start by asking whether this way of writing and posting the workshop outcome was effective, or what else you might try.

Holding this meeting every two to six weeks will allow your team to track its local and evolving culture, to create its own, agile methodology.

The Value of Reflection

The article on Shu-Ha-Ri excerpted in the Introduction continues with the following very relevant discussion of reflection:

"As you learn a technique, and as it asymptotically approaches your mental model of the technique as you see others practicing it, you can begin to reason about the technique. It seems the important questions to ask are:

1. How does this technique work?

2. Why does this technique work?

3. How is this technique related to other techniques that I am practicing?

4. What are the necessary preconditions and postconditions to effectively apply this technique in the combative situation? . . .

"As you develop a reasonable repertoire of techniques that you can perform correctly, you will need to expose yourself to as broad a range of practitioners as possible. As you watch others, you need to ask and answer at least three questions:

1. Which other practitioners do I respect and admire?

2. How is what they do different from what I do?

3. How can I change my practice (both mental model and attempts to correspond to it) to incorporate the differences that I think are most important? . . .

"The questions you need to ask yourself about a competition in your postmortems are:

1. Were you able to control the pace and actions of your opponents?

2. Were you able to keep calm and make your techniques effectively with an unhurried frame of mind?

3. Does your competition look like those of the practitioners you admire?. . .

"Throughout all of this, you must honestly evaluate the results of each 'test'. Cycle back to Shu through Ha and then Ri as you go down dead-end paths."

I couldn't say it better.