Section 12.1. Life Without a Software Process

12.1. Life Without a Software Process

Many process improvement experts see the world as black and white: there are bad software teams without a formal process , and good teams that have one in place. But the world is not that simple. You should understand exactly what a software process brings to the table, and why having a process is important.

12.1.1. Teams Can Be Effective Without a Formal Process

There are many successful teams that do not spend time looking at the big picture and writing down a software process. In most organizations like this, individual programmers or small teams are responsible for entire projects: they talk with the users and stakeholders to understand what they need from the software, they build the software, and they deliver it to the users. People in this situation get used to the "jack-of-all-trades" rolethey do everything themselves, from gathering requirements to designing and building the software to testing and deploying it. This is an effective way to build software: by limiting the number of people involved and making each individual responsible for the entire project, less communication and documentation are required, and there is less overhead and complication for the project.

In addition to the "jack-of-all-trades" situation, there are also "skunk works" programmers who will generally take the initiative and build software that was not necessarily asked for, but that addresses certain users' or clients' needs. This tends to happen when a programmer has a lot of knowledge about the needs of the users, and she can independently identify tasks that are being performed poorly that could be automated with software. When this goes well, the programmer is revered as a genius in the organization: she produced software "under the radar" that was useful to the people in the organization, yet required no work on their part: it was simply delivered, finished, as a wonderful surprise. This is a highly satisfying way for a programmer to workno pressure, no expectations, and high rewards.

Another way that many teams work effectively is by relying on a development manager who has a very good understanding of the needs of the users and stakeholders. A highly capable manager who is willing to put an enormous amount of effort into design, architecture, and programming can serve as the hub of all software projects, parceling out work to individual programmers and integrating it all back together. As long as this person is capable of keeping everything in his head, the team can keep delivering software. This requires an enormous commitment on the part of the manager, but it is also very satisfying and leads to a lot of respect from others in the organization (not to mention job security).

What all these organizations have in common is that they do not have a formal software process. A software process is a set of activities that, if done, will result in software. It's important to recognize that all of the organizations described in the previous few paragraphs do have a software processit's just not a formal, or documented and repeatable, process.

You are already familiar with many of the activities in typical software processes, because they include many of the tools and techniques in the first part of this book! In an organization that has put a lot of effort into a good process improvement program, the process activities will include a wide variety of tasks. These tasks will cover not just programming but also requirements management, project planning, configuration management, quality assurance, and other tasks meant to ensure that the software is of sufficient quality. Making improvements to the process will positively affect the way the team builds software in the future.

It's very important to recognize that teams that do not have a formal process are generally happy! A team that produces successful software projects can point to their successes with pride. The varied work, high visibility, and wide responsibility mean that the programmers are highly respected in the organization. Respected, that is, except when projects fail.

The most common source of failure is that this sort of team does not scale up very easily. While it's true that small projects and small teams can work well without a formal software process, it gets harder and harder to do the work as the scope of the software project gets wider, the team gets bigger, or the users and stakeholders become less available. When a team outgrows an informal process, projects begin to have problems. Programmers who used to produce lots of software find that their projects have started to feel "bogged down." The users are increasingly unhappy and the software is increasingly late. The work just doesn't seem fun anymore. When this happens, many of the "Diagnosing Problems" scenarios in the first part of this book start to look very familiar.

The shift from a happy and productive team to a bogged-down team usually happens because the team has taken on more people, larger projects, or new kinds of work. Suddenly, they have to retain more knowledge about the users' and stakeholders' needs than they can keep in their heads. Some teams find that this happens when they add the fourth (or sixth, or ninth) person to the team; others find that it happens when they try to take on a project that does not resemble any of their previous ones, or when they team people up to work on larger or more complex projects.

One of the most common situations in which a team gets "maxed out" arises when a small programming group with a good track record of building projects for individuals is faced with having to build a project on a larger scaleespecially if it is intended to be used by people outside the organization. Previously, the programmers were able to sit down with the people who would be using the software on a day-to-day basis and talk with them about what they needed. Those users would stay involved with the project throughout the entire development phase, giving feedback on prototypes and intermediate builds and helping the programmers learn more about their needs. A programming team used to working in this environment can have trouble building software meant for users outside of the organization. The problem is that the users are no longer available to the programmer whenever input is needed. A programmer used to walking across the office and asking for clarification about a confusing or unclear concept will often have trouble adjusting to a situation in which that input is not readily available.

This problem is compounded when the knowledge needed to define the behavior of the software does not exist anywhere in the organization. Programmers know how to build software; they are not necessarily familiar with the day-to-day business of the people who will use it. This is where the "jack-of-all-trades" can suddenly find himself on shaky ground. He's spent a great deal of time learning about the particular needs of a small number of users; if a new project requires that he understand a completely different set of users, he may have to spend a very long time building up sufficient expertise in this new area.

12.1.2. Lack of Process Maturity Is Not "Immature"

There are many excellent textbooks on the nuts and bolts of software process improvement, but few of them acknowledge the fact that many successful organizations do not have a formal software process. In fact, process improvement experts often talk about process improvement in ways that are actually insulting to software professionals who feel they have done a good job and gone a long way in their careers. They use words such as "immature," "haphazard," "chaotic," and even "childish" to describe the software process of an organization that does not have a formal software process. This is not a useful way to convince people to change the way they do their jobs. (This was a major consideration in choosing "initial" as the label for the first maturity level in the CMMsee the section "The Capability Maturity Model.")

For example, one otherwise excellent book on practical process improvement contains a chapter demonstrating this problem. It compares the key behaviors of a child (moody, unpredictable, inconsistent, living for the moment, thrown into a panic by unexpected setbacks) with the behaviors of a "mature adult" (stable, reliable, consistently capable, plans ahead, copes well with the unexpected). The author uses this comparison to illustrate the difference between an organization with a formal process and one without. This is quite insulting to anyone who works in an organization without a formal process; it's also untrue, since there are plenty of organizations that in no way display those characteristics. Yet some process improvement professionals attempt to change organizations by showing them exactly this sort of "evidence," presumably in hope of shaming the team into adopting the changes they recommend.

Even worse than insulting people into changing is evangelizing to them, but this is exactly what many process improvement experts do. As unlikely as it sounds, process improvement textbooks and articles literally talk in terms of religion, believers, disbelievers, and evangelism. To someone who thinks this way, process improvement is a way of thinking that requires faith; unbelievers cannot be "converted," no matter how much they are cajoled or harangued. It's not surprising that many process improvement efforts fail. People distrust evangelists and suspect they have more of an allegiance to the process than to the organization and the team. Software engineers who are pestered or pressured into "accepting" a new process will not hesitate to abandon it if they are put under any external pressure to do soespecially if they had no input or feedback when it was developed.

This is no way to convince someone to change the way he does his job. But it is, unfortunately, a representative example of the way some people approach process improvement. They treat the programming team as a bunch of immature children who can barely do the work assigned to them. Or they treat the team members as infidels and unbelievers, leading the team to wonder how these religious lunatics got into the office. Then the experts are surprised that the team memberswho consider themselves and their past projects successful, even if they weren't perfectreject this new process improvement effort.

The truth is that programmers love being the "go-to" person. They love having lots of successes, and they love when users can go directly to them without having to bother with documentation and testing in order to get programs written. There is a lot of job satisfaction in doing things this way. And in cases in which the software is all in the same area of the organization's business and repeatedly draws on the same knowledge of the users' needs, a team can be highly successful working like this. If you put yourself in the programmers' shoes, you'll realize that you would need a really good reason before you would be willing to change. (See Chapter 9 for more information on why changes like this often fail, and how to successfully make changes in your organization.)

12.1.3. If Things Are So Great, Why Change?

If there are no complaints about the way the team is building software, then there's no reason to change! If all your organization needs from you is that you assign one or a few programmers per project, and if you can always guarantee that experts, users, and stakeholders will be available to the programmer whenever she needs them, then it might make sense to keep things flexible and variable and change them only as needed.

However, few teams are really in this situation. At some point, most teams will need to grow in size or capability. This can mean taking on more programmers and solving more complex problems. Without a formal process, organizations often run into problems when they try to expand the team or get people to work together on larger projects. For example, the programmer who was working on one project could be needed on another team, or could leave the organization altogether; this makes it difficult to expand the team because they have lost critical knowledge required to build and maintain the software. If there is no formal process in place, there is no guarantee that knowledge was ever written down.

The most serious problems in expanding a team involve communicating the needs of the organization, users, and stakeholders, and setting the expectations of the organization about what will be delivered, when it will be delivered, and how much it will cost. Having a formal process in place ensures that there is consistency between projects, so an expanding team can rely on a library of past estimates, guidelines for working together, and other important tools to help them work. They do not have to keep reinventing the wheel; for every problem that has been solved before in the organization, the solution is available to them.

Expanding the team is not the only place a formal process is useful. It can also help in an organization where experts, users, or stakeholders are no longer readily available to the programmers. This is another area where the formal process can help. It means that requirements gathering activities are planned for from the beginning, and that there is a training program in place to help the users and stakeholders learn to work with the requirements analysts, ensuring that requirements are gathered at the beginning of the project.