Computing Limits for the XmR Chart

The Tyranny of Control Charts

Every organization I have ever worked for or appraised ends up using control charts. At first, I did not understand why. The reason for my not understanding was because the charts were not generated correctly and the numbers were never explained. It was simply enough to draw pretty pictures that made management happy. One organization I worked for produced 127 control charts. I was called in to help them reduce the number of charts produced. My analysis of the charts revealed one stunning conclusion: although 127 charts were produced, there were really only three charts produced many times over. The charts produced tracked productivity of contractor labor, cost of contractor labor, and CPU uptime versus downtime. The reason so many charts were produced was that some managers wanted them in color ; some managers wanted them for the month ending on the 15th of the month; other managers wanted them ending on the 30th of the month; others wanted them produced weekly; and others wanted them produced monthly and quarterly. But they were basically the same charts! Once this observation was reported to senior management, the lower-level managers were told to only view the charts for data reported at the end of the month, and no more color representations.

Another organization that I appraised did the following "dog and pony show" for the appraisal team. We were given a presentation by the Metrics Team of the metrics and control charts used in this organization. Several members of the Metrics Team stood up, presented their charts on the overhead projector, and then sat down. After the presentations were over, the appraisal team had a chance to ask questions. My team members were silent. They had been suitably impressed by the length and depth of the metrics collection and reporting. The charts were pretty, in color, and seemed sophisticated. I was not impressed. My job was to help my team determine whether the metrics collected were appropriate enough to use to measure the stability and predictability of the processes used in this organization. So I began asking questions of the leader of the Metrics Team. My first question was, "How did you collect the data?" He told me the data were collected using an automated tool. I asked for a demo of the tool. During the demo, I asked, "What data were collected? How was it determined to collect these data? How were the charts used in the organization? What did all the charts show? Where were the data that could back up the information on the charts? What story were the charts really telling?" He could not answer my questions. Instead, he had me talk to the programmer who had coded the program used to collect the data. The programmer could not tell me why the programs used the formulas to collect and calculate data values. He only knew that he was responsible for updating the program. When was the last time he did that? Well, he had been there 12 years and had never updated the program. He told me to talk to the Strategic Planning Manager who was responsible for creating the Master Schedule. So I did. She also could not tell me why these data were selected and what the charts showed. She told me to talk to the Contracting Officer who was in charge of the budget, and billing the clients . She also could not tell me the answers, nor could she explain how the budget was derived or how the billing was calculated. She referred me back to the Strategic Planning Manager. She referred me back to the Metrics Team. Basically, no one in this organization really knew where the data came from, how they were collected, and whether or not they were still relevant. No one had bothered to ask. They were happy producing pretty pictures. No decisions were being made based on these charts.

The two organizations discussed above used control charts to display their data. I soon learned that control charts are a wonderful means of displaying data to use to identify information, when people understand what the data mean and how to interpret the charts. So, while I have titled this section The Tyranny of Control Charts, that is really a misnomer. We will discuss control charts at length.

Control charts are used to identify process variation over time. All processes vary. The degree of variance, and the causes of the variance, can be determined using control charting techniques. While there are many types of control charts, the ones we have seen the most often are the:

• c-chart. This chart uses a constant sample size of attribute data, where the average sample size is greater than five. It is used to chart the number of defects (such as "12" or "15" defects per thousand lines of code). c stands for the number of nonconformities within a constant sample size.

• u-chart . This chart uses a variable sample size of attribute data. This chart is used to chart the number of defects in a sample or set of samples (such as "20 out of 50" design flaws were a result of requirements errors). u stands for the number of nonconformities with varying sample sizes.

• np-chart. This chart uses a constant sample size of attribute data, usually greater than or equal to 50. This chart is used to chart the number defective in a group . For example, a hardware component might be considered defective, regardless of the total number of defects in it. np stands for the number defective.

• p-chart. This chart uses a variable sample size of attribute data, usually greater than or equal to 50. This chart is used to chart the fraction defective found in a group. p stands for the proportion defective.

• X and mR charts. These charts use variable data where the sample size is one (1).

• X-bar and R charts. These charts use variable data where the sample size is small. They can also be based on a large sample size greater than or equal to ten (10). X-bar stands for the average of the data collected. R stands for the range (distribution) of the data collected.

• X-bar and s charts. These charts use variable data where the sample size is large, usually greater than or equal to ten (10).

So, as you can see, you can sometimes use several of the charts, based on type of data and on the size of the sample and the size of the sample may change. While some folks will quote hard and fast rules for the use of these charts, we have found that organizations often modify when they are used and how they are used, based on the preferences of someone influential in the organization. In any case, when using these charts, look for trends or patterns in the data collected. Try to collect 20 to 25 groups of samples to be statistically correct, although five or six may prove useful in detecting initial trends.

The above definitions use the terms "attribute" data and "variable" data. Attribute data are data counted as discrete events or occurrences. For example, Yes/No, Good/Bad, Is/Is Not Defective. These data are usually counts of something. For our purposes, examples are number of CMMI Process Area goals attained, percent of defects found per month, number of trained people on a project team, percent of projects using function points to calculate size. Variable data are data that vary and must be measured on a continuous scale. These measurements are usually quantitative measures. Examples are length, time, volume, height, effort expended, memory utilization, and cost of rework .

Control charts help detect and differentiate between noise (normal variation of the process) and signals (exceptional variation that warrants further investigation). An everyday example of noise in a process is the "white lab coat effect." This effect is what happens when you go to the doctor to get your blood pressure checked. Anyone who has a tendency to high blood pressure will generally become a little nervous during this procedure. Therefore, the blood pressure reading taken by the medical professional (in the white lab coat) has been known to skew higher than if the reading had been taken in the comfortable surroundings of your own home. Another example is for those of us trying to maintain our weight. Even those persons who are successfully maintaining their weight notice fluctuations throughout the month especially women. By tracking our weight over the period of a year, we will find that the average stays about the same. Unless we start super-sizing our meals and indulging ourselves in ice cream or key lime pie. In fact, if we do indulge ourselves several days (or months) in a row, we know that we will gain weight. That indulgence is therefore a signal to watch our intake of food, exercise more, and do all that other stuff we know we should do.