Section 9.1. Why Change Fails

9.1. Why Change Fails

A project manager who is trying to get an organization to build better software must make changes to the way the work is performed. This is easier said than done. Most project managers are not in a position in which they can simply demand that everyone in the organization abandon their old ways and adopt new tools and techniques. But even in an organization where it's possible to simply issue a memo requiring everyone to change the way they do their jobs, people will usually be uncomfortable changing so quickly.

The planning and the execution of the change are very important, and there are aspects of any change under the project manager's control that can cause it to fail. Not all obstacles are political: if you introduce a poor practice or tool, or you implement a tool poorly, it will not go well. When your projects suffer from poor planning, insufficient training, or simple bad timing, your attempt to change your own project may fail.

But there are problems that are bigger than just your project. Sometimes the biggest obstacles to your projects come from outside of what a project manager typically controls. Sometimes there are people impacted by your change who will resist it. If you feel that the change is necessary despite the resistance, you will need to handle it politically.

Most technical people have an aversion to politics. They feel that the only way for one person to influence another is to first lay out all of the facts; the only logical conclusion should be obvious. Unfortunately, when you are working with other people, it may take more than just facts to win them over. Most people who will be persuaded solely by facts will already agree with you. When you encounter resistance to your ideas, it may be from someone whose objection may seem irrational to you. This is especially true when you are trying to make a change that influences someone outside of your project in your organization (like a stakeholder or senior manager) who can prevent your change and who does not necessarily need your project to be successful. To be successful, you will need to understand this person's motivation and persuade him to come around to your point of view. That's politics, and it may be necessary in order for your project to be successful.

Many project managersespecially ones who have a technical backgroundtend to ignore the fact that their organizations are made up of people who need to be convinced of the importance of a change before they will adopt it. Some of these people will have an emotional or even irrational response to any attempt at change; it could take a sea change in the organization before they agree to it.

Irrational attitudes about software development usually boil down to two basic beliefs. First, people believe that most or all software projects are delivered late and delivered with many bugs, and that this is just a fact of life. Second, they believe that their organization is unique, and that the problems they are experiencing are particular to their organization and have never been seen before in any other organization.

(This second belief may seem odd, considering the many thousands of software organizations around the world that have all used similar tools and techniques to fix very similar problems and make real, lasting improvements. It's possible that the belief in uniqueness comes from the fact that the software being built truly is unique, in that it has never been built before; it's not a leap to assumeincorrectlythat the software project and all of its problems are therefore also unique to that particular organization.)

Many times, resistance is not irrational at all. Anyone who has been through a change previouslypossibly a passing management fadthat didn't fix the problem (or failed outright) may be resistant to another change. It may seem unfair, but if people in your organization have previously gone through poorly planned changes, it will be harder for you to make changes of your own.

When you are introducing new tools, techniques, or practices in your organization, you may encounter resistance for a number of reasons. By exploring the feelings, fears, and justifications for resisting change that project managers commonly encounter, these reactions can be unraveled and understood.

9.1.1. Change Is Uncomfortable

Most people in your organization are trying to do a good job. They want their peers and supervisors to see that they are good at performing the tasks assigned to them. When someone has developed a level of comfort and familiarity with his job, the last thing he wants is to have someone come along and make him adopt an entirely new way of doing things.

People can also be afraid of change. Most people prefer to feel like they know what they are doing. They want their jobs to be stable and the tasks assigned to them to remain within their capabilities, and, most of all, they want everyone around them to recognize that they are competent and doing a good job. A new tool or technique may seem like a good idea to you. But if someone in your organization who is affected by your change doesn't understand how it would benefit her, then she will only see and respond to the potential problems that it might create. In other words, it's not enough to show that the benefits of a change outweigh its costs on an organizational level; if people do not see how a new tool or technique personally benefits them, they may see it as a burden with no benefits and will be resistant to adopting it.

Resistance to change is not necessarily a bad thing. For example, when a project's scope or requirements are repeatedly changed, delays and defects usually result. In Chapter 6, a change control process was introduced to make sure that changes are only made after their costs and benefits have been fully explored. If it is important for projects' changes to be controlled, it is even more important that an organizational change is made only after its costs and benefits are well understood. It would be difficult to work in an organization that welcomes any change from anyone, without considering the impact of that change.

It makes sense to be prudent in evaluating the nature of changes made to the way you build software. Too drastic a change can cost you your project. Everybody on the team needs to understand why new practices are put in place, and each person should feel like he understands how his role fits into the big picture. If the way the team builds software is constantly being "improved," it can become too difficult for each person to keep track of his day-to-day tasks, and people can lose their sense of purpose. This can lead to prioritization problems, quality problems, and almost certain delays.

Yet despite the fact that changes can be hard to introduce, you will want to introduce new tools, techniques, and practices. But even when you are fully prepared to show that the benefits outweigh the costs, people in your organization may still resist those changes.

This can lead to a very frustrating situation. You may make a great case for a change: there is a real problem and you have the evidence to support it; many people, both inside and outside your organization, see the problem. The senior managers and the project team all agree that there is a problem. But when you start suggesting changes that address this problem, everyone gets uncomfortable. As each person starts to think about how he would have to change how he does his own job, some will simply start to resist any change. They may resist it because it would require learning a new way of doing things, because it would require them to do more work, or just because it is different.

And even when you convince others of the need for a change, it can sometimes be difficult to prove to the people in your organization that adopting a particular tool or technique will really solve the problem. Often, people will question whether they should go through all of the pain of changing, without knowing for certain that the change will result in a real solution. It can be very difficult in this situation to convince them otherwise.

9.1.2. We Already Build Software Well

Denial is a common response to change. You may have identified a glaring problem, but people around you fail to even recognize it (or simply refuse to acknowledge it). Many professional software engineers and managers have never experienced a project that did not have enormous delays and serious problems; it's often assumed that this is just part of how software is built. After all, they usually delivered somethingmost projects were eventually completed, and the software they built is now being used. Sure, some projects seem to always be eternally 90% done (with 90% left to go), but most of them seem to get something onto the users' desktops (although many patches and bug fixes needed to be rolled out afterward). Isn't that good enough?

The truth is that software development does not have to be painful. It is possible for schedules to come in on time, for teams to work normal hours, and for project managers to feel like they are actually in control of their projects.

The denial mindset should seem familiar to anyone who has tried to implement inspections, only to meet with enormous resistance from the project team. Just as people are not used to seeing their documents criticized and corrected, they are not used to, or comfortable with, their projects being branded "failures." Part of your job as a project manager is to help people in your organization learn from their mistakes. This means that they first need to learn to acknowledge those mistakes.

Sometimes project managers find that people want to know how projects in other organizations have turned out. When people in your organization ask for this, what they are really looking for is the bar that they are being measured against. They don't necessarily want to know what techniques are out there in order to help them build better software; they are just looking for corroboration that they are doing enough already, and don't need to make any changes. But the truth is, projects usually go wrong because of specific problems. Many research groups (including the Software Engineering Institute and the Standish Group) have found that when organizations implement process improvements (like the tools and techniques in this book), those problems often get fixed, leading to software development that costs less and results in better software. Many organizations around the world have verified this in practice.

Also, people in your organization may be willing to acknowledge that there is a problem, only to go back into denial when past performance is discussed. Getting people to talk about their problems is a start, but it's not enough. The same people who complain about project delays and scope creep will often suddenly reverse their opinion when talking about past projects. Most professional people need to see themselves as successfulit's part of each person's identity. For many people, talking about the need for improvement means digging up old, painful projects and examining problems that they would just as soon leave buried. It's easier to look back at a troubled project and see only the successes, forgetting the delays, arguments, difficulties, dead ends, and failures along the way.

In an organization where people are in denial about their problems, it is common for people to shoot down changes by saying that "we've never done it that way before." That's obviously trueit's different, so clearly it hasn't been done that way before; the objection sounds like nonsense. The reason this excuse makes sense to the people who say it is because any change amounts to an implicit criticism of the status quo. If a change is needed, then it means we've been doing things wrong; therefore, any change is bad. Making changes to the status quo seems too risky because the organization has always developed software this way.

Another important reason that changes may seem uncomfortable is that the people in the organization may not fully understand how the software is built. Many senior managers have a great deal of experience with the problems that the software solves: for example, the CEO of a company that builds financial or accounting software knows a lot about finance or accounting. But, frequently, the details of what is involved in building the software are a mystery to them. When senior managers are disconnected from the design and development of the software, they begin to see it almost as a "magic formula." They are clearly making money selling software, so they must have made the right decisions. Now some project manager is coming along and telling them to make changes to this formula, which they don't understand. A senior manager does not like feeling ignorant of his own organization, and he does not like being told that the organization that he built needs to be fixed.

Ironically, in many organizations where people claim that they already build software well, there is no standardized way of building software. Sometimes there are requirements written before the programming begins; other times, there aren't. Sometimes there is a project schedule, but often, there's just a single deadline. In one project, the software may be tested; in the next, it's dumped on the users. Since no two projects are ever done the same way, it's always true that the team has "never done it that way before"and this excuse is still used to shoot down anyone trying to come up with a way to build better software.

9.1.3. "Not Invented Here" Syndrome

"Not Invented Here" syndrome (NIH syndrome) is a name given to a common type of organizational culture where people intentionally avoid research or innovations that were not developed within the organization. When faced with a problem, the people in the organization will typically reject a solution that is known to have worked elsewhere in the industry, solely on the grounds that it did not originate from inside the organization. They opt instead to build their own solution, often at far greater cost.

This may seem ridiculous or silly to people who have not directly experienced it, but NIH syndrome is a serious problem. Some teams will waste many hours defining procedures, creating tools, and building their own solutions to problems that have already been solved elsewhere, rather than adopting or adapting an existing solution that can be purchased off the shelf or learned from a book, training course, or outside expert. One motivator behind NIH syndrome is that people are often rewarded for building new software when they would not be rewarded for buying something that does the same work. For example, a programmer who would get a lot of recognition for spending months building a component might not get any recognition for buying an equivalent one, even though it would cost a tiny fraction to buy rather than build.

If you think about it, you may recognize at least a small example of this behavior in your own organization. For example, many programmers will "reinvent the wheel," building functions or components that could be purchased or downloaded. If your organization commonly develops proprietary technology instead of using an alternative that's available from a third party, it may suffer from at least a mild case of NIH syndrome.

A project manager attempting to change an organization with a bad case of NIH syndrome faces unique challenges. In most organizations, once people recognize that there is a problem, it's often sufficiently convincing to show them that people in other organizations solved the same problem with a solution that already exists. But people in an organization that has NIH syndrome will reject the idea that anyone else in any other organization could possibly have had the same problems. There is simply a pervasive idea that "we're different," which leads to immediate resistance to any ideas that come from outside.

There is an especially virulent strain of NIH syndrome that is commonly found in small entrepreneurial companies. It goes beyond simply believing "we're different"; instead, the people at afflicted organizations believe "we're better." They consider larger companies to be inflexible and laden with bureaucracy. They feel that their small size is an advantage. People in companies like this will often refer to things like their "flat organizational structure" (which, oddly, still features four or five layers of hierarchy, similar to most medium-sized companies), and will talk about how they can respond much more quickly to clients' needs than their larger competitors. Such a mindset often leads to outright rejection of any change based on a tool or technique that works for large companies. Anything that a large company does is dismissed as "too bureaucratic" (see below), and would "clearly" slow down a small, nimble company. In truth, however, every single tool and technique in this book has successfully been used on small projects employing as few as two people. When it comes to NIH syndrome, size really doesn't matter.

9.1.4. It's "Too Theoretical"

When an idea does not make intuitive sense, many people will dismiss it as a result of "academic research," which could not possibly apply to the world they live in. For example, to someone without a project management or software engineering background, it may not be immediately obvious that reviews reduce defects, or that it's important to write a specification before building software. To him, these procedures are time-consuming, with no obvious benefit. They sound good in a book, but would never work in the organization. In other words, they are "too theoretical ."

Some managersespecially ones who consider themselves "hands-on" and who value technical knowledge above management skillwill say that many of the tools and techniques in this book are "too theoretical," and therefore somehow do not apply to their particular organizations. This attitude is common among managers who worked their way up from programming or other IT positions and who have only served as a manager in one organization. It is also common among people who have never experienced a successful project that was on time, on budget, and with few delivered defects.

Declaring a particular tool or technique "too theoretical" may seem like an odd response. These practices arose from software engineering practices that were developed, implemented, and refined over the course of countless projects in thousands of organizations. Most people would consider any solution that is well accepted and has been implemented across the industry to be anything but theoretical. But for many managers, a technique described in a book is of less value than one that was learned or discovered in the field. If someone has never seen a project that went smoothly, it seems natural to assume that all projects suffer delays, scope creep, and other problems that stem from poor planning and engineering; they consider any proposed "cure" for these problems to most likely be nothing but an academic theory that would never work in the real world.

It is especially common to hear the "too theoretical" excuse from the manager or leader of a small team facing growing pains. This typically occurs when programmers are added to a very small team. The team slowly gets less and less productive, and adding more programmers does not seem to help. Many of the problems described in Part I start to occur, and the manager gets frustrated. Unfortunately, this is a very difficult (yet common) position for a new manager to find himself in. Typically, he finds himself trying to doing everything the same way it's always been done, but his projects continue to break down. It seems like every trick he's tried has backfired. He's called big meetings that turned out to be worse than useless. He's yelled at his team, other managers, even the CEO. It seems that everyone knows that something is wrong, yet nothing will fix the problem.

Still, this manager will often reject tools and techniques like the ones in Part I as "too theoretical" because he has not personally experienced them. When there were only two or three people on the team, the projects apparently went just fine. It seems intuitive to him that adding a few more people should not make much of a difference in managing the project. To this person it's not intuitively obvious, for example, that writing a vision and scope document or creating a project schedule will help his projects run more smoothly. Anything that he has not tried, and especially anything that does not directly affect the code, must be "too theoretical."

The problems that are diagnosed in the first part of this book really do affect very small teams; they are just not nearly as difficult to overcome when only two or three people are affected. It is not hard for one person to keep track of changes, communications, and project status for a very small team. However, as the team grows, these problems compound. The team starts to lose track of changes to the scope, requirements, and code. The project schedules get more complex and less accurate. Stakeholders and users start to feel that the project is getting out of control.

A manager in this situation will be very cautious about adopting any change. He knows that just making changes at random makes the problem worse. He's probably read management books and tried applying the information in them, only to find that didn't work; he's come to distrust this sort of advice. By demanding changes that are not "theoretical," he's really saying that the team must be careful not to implement any more changes unless they are already provenpreferably on one of his own projects. Unfortunately, that creates a chicken-and-egg situation: how can the team take a particular technique from theory to practice, when every technique is too theoretical to be implemented?

9.1.5. It Just Adds More Bureaucracy

An especially damaging attitude in some organizations holds that programming is the only important activity in a software project. Project management tools and techniques are seen as distractions that drain energy and effort away from the programmers' "real job" of writing code. Any project activity that takes place before the programmers begin writing code simply amounts to "spinning our wheels," and the goal of all early project activities should be to get the programmers doing the "real work" as quickly as possible.

A manager in this sort of organization will typically think that if she just adds more programming hours, a project that seems to be failing will get back on track. Since programming is the main activity in the project, all project problems amount to programming problems. In fact, there are project managers who believe that all they need for a successful project is a team of top-notch programmers. They wonder, "Why do we need to quadruple our documentation by creating schedules and project plans and change control procedures?" After all, it's been shown time and time again that a great programmer is 10 times as productive as a mediocre one. Isn't it enough to pay top dollar getting the very best programmers and setting them loose on the problem? Why all the extra "bureaucracy"?

In reality, programming is usually less than 40% of the effort on a successful project. Most project problems are caused by the team not understanding what it is that the software should do. Estimation problems happen when the team members don't explore all of the assumptions that they are making and, as a result, don't have a handle on what information is known and what is unknown. Project planning goes wrong when the scope creeps, or when there are problems that could have been foreseen. When bugs are found in the software, often it's not because the software is broken; the software is usually working exactly as the programmer intended it to work, it's just not doing what the users need it to do.

The solutions to these problems do not involve programming. But, to some people, any action that the team takes that does not directly relate to programming is "bureaucratic." Planning the project, writing down requirements, and holding inspection meetings is seen as just pushing paper around. There may be a project schedule, but this is just used as a tool to help the rest of the organization understand what it is the programmers are doingin other words, the schedule is created by asking the programmers their opinion, and the project manager's job mostly boils down to simply reporting that opinion to the rest of the organization. (Even worse, of course, is the project manager who simply makes up the schedule out of thin air, or bases it solely on the boss's expectations.)

When the project manager's role is reduced like this, there is usually an implicit assumption that a schedule is just a guess, and an expectation that it will almost certainly slip. At least the project managers and other software engineers believe that it's implicitly assumed. Unfortunately, this is a dirty little secret among the engineering team. The rest of the organization takes these estimates and schedules at face value. They believe that the team made a real commitment, and they make important decisions based on what they believe is a real date. This leads to a general feeling of distrust between the software engineering group and the rest of the organization. It all starts with the project manager's mistaken attitude that it's just not possible to estimate a project accurately, and that you just can't predict what's going to happen when the team starts building the software. (See Chapter 3 for more information on how to estimate a project accurately.)

In an organization where this attitude is prevalent, it is common to see a much higher value placed on anything that has to do with programming, with a lower value placed on all other areas of software engineering. If the programming team requests new computers, often senior managers will not hesitate to spend thousands of dollars on them. Yet the project manager may be refused permission to spend a few extra hours talking to stakeholders and users in order to write a vision and scope document, even though it would save an enormous amount of project time (and only cost the equivalent of a few hundred dollars of people's time). When this happens, it's usually because it is easy to justify any activity or expense that is done to benefit programming, while all other expenses are regarded suspiciously.

9.1.6. You Can't Give Me More Work!

Most of the changes that a project manager makes will increase the workload of other people. Software engineers, managers, and stakeholders who were not directly involved in building software will suddenly find themselves expected to attend status and review meetings, participate in planning and estimation activities, work with someone creating a vision and scope document or eliciting requirements, or perform other tasks that were never expected of them before. If you are making these changes, then you are the person piling additional work onto someone's already overflowing plate. Not surprisingly, there are people who will not be happy with this arrangement.

People are often unhappy to be asked to attend meetings, especially in organizations where most meetings lack direction, focus, or even an agenda. In organizations like this, the meetings tend to meander. Often they boil down to a discussion between two people about a topic that has nothing to do with anyone else at the meeting. Sometimes, meetings are called simply to give a captive audience to a senior manager. If you are in an organization where everyone hates going to meetings, when someone suddenly finds out that she has to attend your weekly status meeting, she might be unhappyand provoked into doing something about it.

A common response from a recipient of this new workload is surprise and shock. It's very uncommon for people to have tasks assigned to them by people to whom they do not report, yet here's some project manager doing exactly that. Often, when someone is surprised by extra work, he feels especially motivated to take action to remove it from his plate. He will complain to his boss and to his boss's boss, and he may even try to go over your head to get your entire project shut down. When a project manager tries to put new tools or techniques in place, it is frustrating to encounter resistance from the very people who will benefit from the practices simply because they seem like extra work. (It is especially frustrating when team members agree in principle that those practices will reduce the total effort required to build the project, yet do not agree to adopt them!) Even if you win the battle and get your senior managers to agree to force him to do the extra work, you could still lose the war: now a participant in your project cannot be relied upon, and could cause damage to your project by stonewalling and causing delays. He could even sour your organization to any future improvements if the senior managers hear only his complaints and do not see immediate results.

9.1.7. It's Too Risky

The celebrated economist John Maynard Keynes once wrote, "Worldly wisdom teaches that it is better for the reputation to fail conventionally than to succeed unconventionally." Legendary investor Warren Buffett put it this way: "Lemmings as a class may be derided but never does an individual lemming get criticized." In other words, any manager who backs a change puts his reputation on the line; if that manager does nothing, he will not be criticized for maintaining the status quo.

When you make a change in an organization, you aren't just altering the activities that the team performs. You're also affecting how people relate to each other in the organization. A change that goes wrong can ruin someone's reputation (and, if it's serious and public enough, their career).

Nobody gets blamed if things stay the same. If a project fails, but many other projects failed in the same way, the failure is usually seen as inevitable and the people in charge of the project are not held accountable for the failure. And even the most flawed deliverable can be painted as a success, even if the software that is finally produced barely does what it's supposed to and requires many patches and bug fixes. (In fact, for many software professionals, this is a fact of lifeevery project they have ever worked on turned out that way!)

Many people equate questioning what has always been done with insulting the organization as a whole. They consider it to be something a team player would never do. To someone with this attitude, the fact that you are pointing out problems and suggesting changes sets you apart from the accepted culture. They feel that since you are making wide-ranging criticisms of the organization, you do not buy into its culture and are simply being counterproductive. Paradoxically, to these people, it is considered more productive to make bad decisions and let projects fail than to question the way things are done.

This means that when people complain that making changes is risky, they are not incorrect. However, a manager shooting down a proposed change as "too risky" may be sensing risk to his own reputation, rather than to the project.

When looked at from a pure cost-benefit perspective, most of the tools and techniques in this book have a very limited risk to the project. Usually, it only takes a few hours to run a Wideband Delphi session, set up a version control system, write a vision and scope document or hold an inspection meeting. Even if these tools and techniques fail to produce any results at all, the total cost to the project is minimal. Yet these same tools are routinely shot down as "too risky," and in a sense they arebut not with respect to the project.

When people talk about these tools and techniques being risky, often they are not trying to say that the tools will somehow damage the project; rather, the risk is that if the project still runs into problems, the manager who implemented the new tools can be held responsible for those problems. In other words, there's an implied rule in most organizations: "you break it, you bought it." If you make a change to an existing process, you're now responsible for any failures that result, even if those failures are only tangentially related to the changes you made.

People are justifiably reluctant to go against the conventional wisdom in their organization. No matter how misguided, the popular opinion is very powerful and difficult to change. People who are willing to point out a series of singular, unrelated mistakes may be unwilling to admit that there is a more general problem with their organizationespecially if admitting the problem calls into question their knowledge of software development. People do not like to feel like they don't have all the answers. And if you point this out in your effort to change your organization for the better, you might inadvertently cause people to feel that you are publicizing their mistakes and calling them out as incompetent.

Incidentally, this is one reason many people are turned off by the idea of process improvement. They have only seen attempts at change partway through a project that has gone awry. Managers start making out-of-context changes in desperation, usually based on something they have read but don't fully understand. Even worse, the changes are frequently used as a means of forcing a stakeholder or client into compliance with an already delayed schedule. This is not so much a way to save the project as it is a way for some manager to protect his reputation. Not all improvements will help in every situation.

For example, one of the most common places where changes should not be introducedbut often areis when a project is starting to fail and the stakeholder is starting to get angry. This mistake is especially common in consulting companies that gave unrealistically low estimates for their projects and now find themselves in trouble. They see a client who rightly points out that the software has many defects, and that it does not fulfill his needs. A common (and unfortunate) response from the project manager is to try to change the organization, instead of admitting that the client was given an unrealistic plan and working to better fill the client's needs. A change control process is implemented to clamp down on any new feature requests, and a hard-and-fast schedule is put in place that forces the client to accept the work that the team has already done, whether or not it fulfills the client's needs. This results in a frustrated client and increased friction between the client and the project team.

Forcing a stakeholder to accept poor software is a terrible reason to make changes to the software process. Organizations are right to resist changes that are made for any reason other than building better software more quickly. If there are client or stakeholder relations problems, they need to be dealt with directly, instead of trying to solve those problems by changing the way the group builds software.