Extreme Maintenance

The Orbix Generation 3 Engineering and Management team realized that XP represented a model for maintenance and enhancement that fit very well in the context of the process improvements that we were initiating at the time. Kent Beck himself notes that "maintenance is really the normal state of an XP project" [Beck2000]. Both managers and engineers felt that this statement was correct.

In our earlier projects, we had created a set of maintenance processes that described how a bug was created, prioritized, analyzed, fixed, and delivered to the customer. The Generation 3 team also had a set of processes that described how incremental enhancements were requested, described, prioritized, implemented, and delivered to customers. It was clear in looking at these processes that they already incorporated important elements of XP. The synergy was so great that we started to use the term "Extreme Programming" to describe the set of processes and practices used by the Generation 3 element and product teams. We started the XP experiment in earnest by looking at the continuing problems mentioned earlier to see how some of the XP approaches might help us improve.

The Stories

When customers report a bug, they provide a description of the bug in the form of a short "story" that describes what they are trying to do and how the problem manifests itself. This story is stored as a note in our defect tracking system and is available to everyone working with the customer, including the customer.

When we get an enhancement request or a functional requirement, we currently provide it as a requirement specification on our requirements Web page. Because of the need for global visibility and the ability to update globally by customer services and engineering, these requirements are not in the form of index cards. They also do not read as stories (yet), but they do contain essentially the same information.

For the purposes of our XP experiment, we left the enhancement requests until a later improvement phase and instead focused on ensuring that the bug-fixing stories were being tracked in a well-structured Web-based queuing system that was visible to the internal customer (customer services) as well as to engineers and engineering managers.

Task Cards and Storyboards

We liked the idea of using index cards to track tasks and decided to implement a storyboard for each of the element teams that made up our consolidated Generation 3 team, to try to fix the problem of poor visibility of each engineer's work effort. Each task, regardless of whether it is a customer issue or an internal project task, is written on a color-coded card (see Figure 20.1), which is placed on a storyboard. The Generation 3 operations lead sits down with our customer (customer services representative), and the customer prioritizes all the customer tasks. The customer also has visibility of internal tasks and may at their prerogative ask to have customer tasks escalated above internal tasks. Currently, only the customer issues are rigidly prioritized, but we intend to integrate internal tasks into that prioritization process as well.

Figure 20.1. Task card

An engineer provides an estimate for completion on the card once they have had 24 hours to do some initial analysis. When the task is completed, the actual date of completion is recorded. If there were delays in completing the work, the engineer simply notes on the back of the task card the facts about the delays, their observations concerning the delays, and some lessons learned. Cards for all closed tasks are removed from the board, data is extracted from the cards and put in a spreadsheet, and the task cards are stored in a task log.

Stand-up Meetings

One aspect of XP that was also incorporated into the team was the daily stand-up meeting (see Figure 20.2). Each team in the group takes 15 to 30 minutes to review their progress. Each person takes a couple of minutes to talk about what they did the previous day and a little about what they are doing today. Again, the focus was on improving visibility but also on encouraging communications between team members who may not necessarily (it's not part of the culture yet) talk to each other regularly.

Figure 20.2. Stand-up meeting at the storyboard

The qualitative results were immediate. Some of the engineers found it difficult to explain to peers why they were spending so much time on issues or why they were looking at things that weren't high priorities on the team's list of tasks.

Testing

One of the first initiatives we undertook in 1999 was to automate our test suites and speed them up. We completed an extension of that effort in 2000, enabling developers to build and test the entire product set on any platform combination we support by clicking a single button from a Web-based interface or using the workstation command line. These initiatives enabled us to start testing the entire product set against the CORBA specification, against the Orbix API, and against customer-supplied test cases, nightly or at the engineer's discretion, on all 17 platforms that we support.

Both the stories describing the functionality supported by the software and the stories describing the bugs that the customers encountered have tests associated with them. Engineering requires that each customer provide a test case for each reported bug before doing any work on the issue. This test case is automatically included in the nightly runs of the test suite. Again, in hindsight, the processes that we decided to follow map very well to the ideas about testing, which are the foundation of the XP model: Test before you implement, and test everything often.

Pairing

First Experience

Our pair programming experiment came about accidentally late in 2000. The Generation 3 team worked on a particularly difficult set of customer issues. We had a customer engineer working on-site with us. (Having customers on-site is another principle of XP, although normally we treat our on-site product managers and customer service representatives as our customers.) At different times, the customer teamed with various members of the Generation 3 team to do some pair programming. Likewise, members of the team gravitated toward a pair programming approach as they progressed with the issues they were working on for this particular customer.

The Results

In the end, the qualitative feedback from the team was very positive. They felt that they worked much more effectively and that there was a much higher level of communication when they worked in this way as opposed to working independently on individual issues. The senior staff also noted that they enjoyed some of the mentoring that naturally occurred as they interacted with the junior members of the group.

The overall productivity during this period was also high. Figure 20.3 shows that for the weeks from November 16, 2000 to December 7, 2000, on average, approximately 67% more issues were fixed than in previous periods. This higher productivity registers as a bump on the eight-week running average. Granted, not a lot of data is there to back it up, but when combined with all the positive feedback, it was definitely felt to be significant.

Figure 20.3. Team bug-fixing productivity

The bottom line is that morale improved. Engineers got a lot of positive reinforcement from each other. Engineering practices and quality improved as we actively noted coaching interactions between engineers. We also got no patch rejections for the work these engineers did.

Team Environment

This pair programming could not have happened without the changes that were made to the engineers' workspace. Before initiating our XP project, IONA's engineering workspace consisted of the typical floor plan with multiple bays of six shoulder-height cubicles in each bay (see Figure 20.4).

Figure 20.4. Orbix Generation 3 cubicles

The engineers and engineering managers initiated a project to restructure the space they worked in to enhance team communication. A large central area was created by eliminating 20% of the individual cubicles and establishing nondelineated group workspaces. The group workspaces consist of either a round or rectangular meeting table or workstation area. In addition, whiteboards and flipcharts on tripods were made available in each work area. A couch was purchased, and catering services bring in snack food (fruit and so on) weekly (see Figure 20.5).

Figure 20.5. Orbix Generation 3 new team area

The results are very noticeable. Code reviews are happening in the group area; pair programming is happening at the workstations. People are discussing their ideas on the whiteboards and large flipcharts. There is a measurable increase in the visibility of what everyone on the team is doing and in the number of conversations and collaborations that are occurring. These changes to the environment were instrumental to our success in improving overall team morale. We also have had a visible increase in the number of team interactions on code reviews and have seen many more discussions in the group areas about ongoing tasks. It is felt that these changes have also improved visibility on tasks.

Feedback and Refactoring

Rapid Feedback

As well as describing functionality, XP focuses on increments of functionality that can be merged into the mainline rapidly. Bug fixes are naturally merged in quite rapidly, and with the improved nightly build and test and the one-button test, systems can be made available as a temporary or permanent patch within one day. We also can see how healthy the mainline is daily across all platforms. Because we have just started collecting metrics on mainline breakage, we don't have any statistics to show what is currently a qualitative analysis of the improvement.

Refactoring Improves Maintainability

Because we were working with a legacy code base, we did not have much to say about how it was put together. However, we needed to make it maintainable and add functionality if required. As mentioned earlier, one of the improvement efforts we undertook was a major refactoring of elements of the Orbix product. This refactoring effort incorporated stories or functionality implementation that typically lasted three to four weeks. By keeping these increments relatively short, we could more effectively manage the delivery schedule of our last minor point release by modifying the scope of the project because we were almost always at a point where we could deliver a functional product. We achieved this without knowing Beck's approach, and in hindsight we recognized that what we achieved was in fact an implementation of XP principles.

We continue to monitor problem modules using our defect tracking system, and using our code reviews, we have started to see refactoring become part of the engineers' toolkit when they approach a problem. The benefit to all this is that we are seeing far fewer problems reported against our new releases. In addition, the code entropy has decreased significantly as measured by the comments from code reviewers.