The Say-Do Trap

During the 2002 U.S. elections pundits, politicians, and the public were shocked by the wide discrepancy between polling data and the outcomes in many key races. Dozens of contests described by pollsters as too close to call ended in a sound drubbing of one of the candidates. The political parties and news organizations that had paid royally for measures of voter sentiment during the campaign were left scratching their heads and wondering what they had gotten for their money. What those politicos and newspeople failed to recognize is that there is often a big difference between what voters say to preelection pollsters and what they do in the voting booth. The same say-do disparity exists in the workplace among both employees and companies.

Like political pollsters, companies that rely on surveys and exit interviews to understand what their employees value and what motivates their behavior often fall victim to the say-do trap. The reason is simple: What employees say is not always what drives their behavior. For example, a person in an exit interview may say that she is leaving because of a higher-paying offer elsewhere, but is that statement a reliable basis for action? Many defectors make this statement because higher pay is a socially acceptable reason to leave. Better still, it will not be challenged and will not make anyone think less of them. In reality, the employee may be leaving because she thinks the company’s managers are dimwits or because the workload is excessive. These are the things that actually undermined her commitment to the organization and made her receptive to other possibilities. However, she won’t say this for fear of burning her bridges, creating bad feelings, or admitting to something that may be interpreted in an unfavorable light. Who, after all, would say, “I’m leaving because the work is too demanding”? Moreover, it is common for employees to load onto a single factor, such as pay, the complex set of factors that influence how they perceive their jobs and employers. That by no means signifies that that single factor is what the employee most values or is most predictive of actual behavior. It is simply a convenient proxy for the entire employment package. Whatever the reason, in these circumstances the company will conclude erroneously that it may not be paying enough for talent. We’ll explain below how FleetBoston Financial avoided that trap.

Companies invest millions of dollars every year in employee surveys to understand what employees value, what they like and don’t like about their group and organization, what keeps them on the job, and what would induce them to leave. However, few companies look at the actual record of stay/leave decisions: what employees “do” and the context in which those decisions are made. Few measure the objective antecedents of turnover and use what they learn to anticipate employee responses to policy changes.

This omission is mystifying in light of the way in which marketing personnel in many companies approach customers. Good marketers take a multidimensional approach to understanding customer preferences. Yes, they ask. They conduct surveys, run focus groups, and do all sorts of tests to determine what customers value in their products and services; what product and service characteristics appeal to customers the most; how they would respond to changes in price, quality, and product and service design; and so forth. They do all these things, but they also track and measure actual buying behaviors: what customers do. Point-of-sale data capture in retail stores is just one example. Those data speak loudly and clearly about customers’ choices. In short, smart marketers consider both what customers say and what they do. They understand the distinction between “stated” preferences and “revealed” preferences and recognize the value of examining both. They follow the dollar trail and use actual buying patterns to draw inferences about customers’ values and to anticipate customers’ responses to price and other product changes. They use that information to make decisions about what they will bring to the market and how they will price their products and services.

Benetton, the apparel maker, has been a pioneer in using that customer behavior approach. Instead of just asking which colors customers prefer, it uses its point-of-sales information system to capture the colors of actual purchases on a daily basis and sends the data to the factory floor, where undyed stock is ready for the finishing steps. Within a matter of days those items—in the preferred colors—are on the shelves and in the shopping bags of Benetton customers. In today’s information-rich world, what marketer would dream of asking customers what they thought without looking also at what they bought?

Somehow the marketer’s approach to obtaining customer knowledge and anticipating customer behavior seldom finds its way to the realm of decision making about human capital, where knowledge of employee values and preferences is based almost exclusively on what the employees say. Employee surveys are everywhere, but what employees actually do is ignored when people policies are developed despite the fact that companies have vast, easily accessible information on employee behavior. The following case is typical and underscores the potential costs to companies that fail to dig deeper for the facts.

Toyota Avoids the Say-Do Trap

Toyota Motor Manufacturing North America almost fell victim to the say-do trap not long ago. That world-class company conducted an annual employee survey and relied heavily on it in formulating its human capital programs. The company believed that pay and promotion were tied closely to employee performance. It also offered extensive training and career management designed to improve skills and give promising individuals opportunities to broaden the scope of their know-how through moves within the company.

Toyota naturally was surprised when survey data revealed that employees discounted both of those expensive programs. In the collective view of employees pay and promotions were not tied to performance, and employees did not see any payoff from training and movement around the company. In light of what the employees said, Toyota was wasting millions of dollars each year on those programs. Once they heard the voice of their employees, company executives considered revamping the performance management and compensation systems, with spending shifted to other programs.

It turned out, however, that there was a major disconnect between employee perceptions and company practices. An analysis of who actually received higher pay and promotions indicated that rewards indeed went to high performers. Payroll and human resources (HR) records provided the proof. Contrary to the employee survey data, pay and promotions consistently were tied to performance in what was an elegantly functioning system. The same thing was true with respect to training and movement within the company. An examination of HR records over several years indicated quite clearly that all else being equal, the people who completed company-sponsored training and those who accepted moves within the organization were being promoted at much higher rates than were those who didn’t.

If this company had relied entirely on what its employees were saying, it would have wasted lots of time and money changing or eliminating human capital programs that were essentially sound. However, if the company had not listened through the survey and instead had relied only on HR information systems and payroll records, it would have missed the real problem: a lack of understanding among employees about what the company actually valued, what criteria were used to evaluate performance, and how specific behaviors and results were translated into rewards. The insights came from pursuing both say and do data. What Toyota really needed was a relatively inexpensive revamping of its communications with employees. The company needed to communicate the facts about performance, training, and movement and their connection with higher pay and promotion. It could then use those facts to bridge the gap between management and employee perceptions (Figure 2-2.)

Figure 2-2: Solving the Say-Do Problem

How to Avoid the Trap

Employees’ stated preferences for benefits arrangements, training options, and so forth, do not always reflect how those individuals will behave when faced with real-world trade-offs. They are facts, but not the complete facts for decisions about human capital. This discrepancy often results from the way surveys are designed. For example, a survey that asks employees to rate the importance of pay in their decisions to stay or leave will not always reflect an individual’s future behavior. When there is no cost associated with rating a benefit highly, respondents generally will say that that benefit is “very important.” The real world, however, requires people to make trade-offs between things they value, even things they value highly, such as higher pay and more time with their friends or families. Everything cannot be “very important.” In a world of constraints people give up one value to have more of another, revealing their highest preferences through their behavior.

Market researchers have known this for a long time and employ techniques such as conjoint analysis to clarify those trade-offs. Unlike surveys that offer “not important,” “important,” and “very important” as choices, more modern sensing methods pose questions in a more realistic context that takes into account the fact that limited resources require people to make trade-offs. These methods force choices and allow the researcher to draw inferences about the value of the individual components of a complex offering. Those techniques have been adapted to researching employee preferences. For instance, one can examine how employees with similar characteristics value different components of a pay and benefits package and then use that information to fashion a package that delivers higher utility to employees, perhaps at lower cost to the employer. The value of doing this is clear, but a basic limitation remains: At best, such methods offer better ways to mimic actual behavioral scenarios, but they do not capture actual behavior. The difference is often significant and extremely revealing about the workforce and the organization.

The say-do disparity does not pertain only to employees. Companies also say one thing but do another, usually without realizing it. One well-known high-technology company, which we’ll call Digitt, thought of itself as a pay-for-performance organization, offering multiple bonus programs, profit sharing, recognition awards, and various broad-based stock plans. When we read the documents describing the various reward programs, we concluded that pay for performance was certainly the foundation of that company’s reward philosophy, but that was what the company said. What did it actually do?

To answer the question, we followed the money trail, evaluating over time who did well in the company and what kinds of performance were rewarded. We discovered several important facts:

• Individual performance was not differentiated substantially. Only 5 percent of total pay was affected directly by individual performance.

• Performance pay was channeled to business units. Lower performers shared equally in the bonus pool by virtue of being in a strong group, receiving over $30 million in the year in which we studied the situation and almost $100 million in total “performance” pay.

• Companywide, people in the lowest quartile of performance were getting 25 percent of all performance pay, more than those in higher quartiles were getting and almost as much as was being received by the company’s highest performers (Figure 2-3).

  
  Figure 2-3: Significant Variable Pay Went To Low Performers

• A low performer in a high-performing group could expect a larger dollar bonus than could a much higher-performing individual who happened to work in the newer business units that had not yet reached profitability.

Without realizing it, this company was shelling out about $13 million each year to employees who had been ranked in the bottom performance quartile for five years in a row: about 6 percent of the employee population.

The bottom line was that the company basically rewarded two things: years of service and business unit performance. It mattered more where employees were in the organization than how they did. Focusing on what the company actually rewarded, not its stated policies, made it possible to cut through the say-do problem and identify millions of dollars in misdirected rewards. Those poorly targeted investments were costly, undermined the workforce, and jeopardized the company’s competitive position. In the end investors turned on the company with a vengeance, destroying over 50 percent of its share value.

The Noise Problem

Capturing facts at a specific moment in time presents another problem for decision makers: the effects of random environmental influences, or noise. In the parlance of statistical analysis, noise is the transitory and unexplained variability found in any set of data. If there is significant noise in snapshot data, decision makers can be led astray.

Consider this example. The sales revenue of a bookstore fluctuates from month to month, week to week, and day to day. A substantial portion of that variability is based on known market factors: the seasonality, the year-end holidays, day-of-the-week shopping patterns, and so forth. These variations can be adjusted out of the data. Another part of the variability reflects the actual performance of management and company employees: what they actually do. The variability that remains is noise. Noise in the bookstore’s revenues might be caused by temporary road construction near the store, by a large one-time order, or by unknown causes. Thus if one were evaluating the bookstore’s performance on the basis of a short-term snapshot, the facts could be skewed by those transitory events.

The remedy for noise is to look at facts over time, smoothing performance to get a clearer picture of the more persistent patterns of performance. This is something that people do in most business analyses but seem to overlook in getting the facts about human capital.

Our prescription for avoiding the say-do trap is to get the “right” facts by rigorously capturing perceptions and observing actual behaviors and company practices, the way Toyota does. In looking at the “say,” consider the perceptions of both employees and the company itself. Then uncover actual behavior—the “do”—from company records and the behavior of employees in different environments over time. Recognize that learnings derived from the analysis of hard data have limitations as well, since not all programs or events with which the organization and its employees experience are recorded in electronic files. Sometimes it is impossible to find a reasonable proxy or indirect indicator for a program or event of interest. In such cases, there is nothing better than to ask. Once you have the say-do facts, you can assess their business relevance by means of statistical modeling.