PLANNING THE EXPERIMENT | Six Sigma and Beyond: Design for Six Sigma, Volume VI

The purpose of this section is to:

Impress upon the reader the importance of planning the experiment as a prerequisite to achieving successful results
Present some tools to use and points to consider during the planning phase
Demonstrate DOE applications via simple examples

BRAINSTORMING

The first steps in planning a DOE are to define the situation to be addressed, identify the participants , and determine the scope and the goal of the investigation. This information should be written down in terms that are as specific as possible so that everyone involved can agree on and share a common understanding and purpose. The experts involved should pool their understanding of the subject. In a brainstorming session, each participant is encouraged to offer an opinion of which factors cause the effect. All ideas are recorded without question or discussion at this stage. To aid in the organization of the proposed factors, a branching (fishbone) format is often used, where each main branch is a main aspect of the effect under investigation (e.g., material, methods , machine, people, measurement, environment). The construction of a cause-and-effect (fishbone or Ishikawa) diagram in a brainstorming session provides a structured, efficient way to ensure that pertinent ideas are collected and considered and that the discussion stays on track. An example of a partially completed cause-and-effect diagram is shown in Figure 9.1.

Figure 9.1: An example of a partially completed fishbone diagram.

After the participants have expressed their ideas on possible causes, the factors are discussed and prioritized for investigation. Usually, a three-level (high, moderate, and low) rating system is used to indicate the group consensus on the level of suspected contribution. Quite often, the rating will be determined by a simple vote of the participants. In situations where several different areas of contributing expertise are represented, participants' votes outside of their areas of expertise may not have the importance of the expert's vote. Handling this situation becomes a management challenge for the group leader and is beyond the scope of this document ” the reader may need to review Volume II of this series.

During the brainstorming and prioritization process, the participants should consider the following:

The situation ” What is the present state of affairs and why are we dissatisfied?
The goal ” When will we be satisfied (at least in the short term )?
The constraints ” How much time and resources can we use in the investigation?
The approach ” Is DOE appropriate right now or should we do other research first?
The measurement technique and response ” What measurement technique will be used and what response will be measured?

CHOICE OF RESPONSE

The choice of measurement technique and response is an important point that is sometimes not given much thought. The obvious response is not always the best. As an example, consider the gap between two vehicle body panels. At first thought, that gap could be used as the response in a DOE aimed at achieving a target gap. However, the gap can be a symptom of more basic problems with the:

Width of the panels
Location holes in the panels
Location of the attachment points on the body frame

All of these must be right for the gap to be as intended. If the goal of the experiment is to identify which of these has the biggest impact on the gap, the choice of the gap as a response is appropriate. If the purpose is to minimize the deviation from the target gap, the gap may not be the right response. A more basic investigation of the factors that contribute to the underlying cause is required. Do not confuse the symptom with the underlying causes. This thought process is very similar to the thought process used in SPC and failure mode and effect analysis (FMEA) and draws heavily upon the experience of experts to frame the right question. In DOE, the choice of an improper response could result in an inconclusive experiment or in a solution that might not work as things change due to interactions between the factors. An interaction occurs when the change in the response due to a change in the level of a factor is different for the different levels of a second factor. An example is shown in Figure 9.2.

Figure 9.2: An example of interaction.

The choice of the proper response characteristic will usually result in few interactions being significant. Since there is a limitation as to how much information can be extracted from a given number of experiments, choosing the right response will allow the investigation of the maximum number of factors in the minimum number of tests without interactions between factors blurring the factor main effect. Interactions will be discussed in more detail in Section 3. The proper setup of an experiment is not only a statistical task. Statistics serve to focus the technical expertise of the participating experts into the most efficient approach.

In summary, the response should:

Relate to the underlying causes and not be a symptom
Be measurable (if possible, a continuous response should be chosen )
Be repeatable for the test procedure

The prioritization process continues until the most critical factors that can be addressed within the resources of the test program are identified. The next step is to determine:

Are the factors controllable or are some of them "noise" beyond our control?
Do the factors interact?
What levels of each factor should be considered?
How do these levels relate to production limits or specs ?
Who will supply the parts , machines, and testing facilities, and when will they be available?
Does everyone agree on the statement of the problem, goal, approach, and allocation of roles?
What kind of test procedure will be used?

When all of these questions have been answered , the person who is acting as the statistical resource for the group can translate the answers into a hypothesis and experimental setup to test the hypothesis. The following example illustrates how the process can work:

A particular bracket has started to fail in the field with a higher than expected frequency. Timothy, the design engineer, and Christine, the process engineer, are alerted to the problem and agree to form a problem-solving team to investigate the situation. Timothy reviews the design FMEA, while Christine reviews the process FMEA. The information relating to the previously anticipated potential causes of this failure and SPC charts for the appropriate critical characteristics are brought to the first meeting. The team consists of Timothy, Christine, Cary (the machine operator), Stephen (the metallurgist), and Eric (another manufacturing engineer who has taken a DOE course and has agreed to help the group set up the DOE).

In the first meeting, the group discussed the applicable areas from the FMEAs, reviewed the SPC charts, and began a cause-and-effect listing for the observed failure mode. At the conclusion of the meeting, Timothy was assigned to determine if the loads on the bracket had changed due to changes in the components attached to it; Christine was asked to investigate if there had been any change to the incoming material; Stephen was asked to consider the testing procedure that should be used to duplicate field failure modes and the response that should be measured, and all of the group members were asked to consider additions to the cause-and-effect list. At the second meeting, the participants reported on their assignments and continued constructing the cause-and-effect (C & E) diagram. Their cause-and-effect diagram is shown in Figure 9.3 with the specific causes shown as "C1, C2, ..." rather than the actual descriptions that would appear on a real C & E diagram.

Figure 9.3: Example of cause-and-effect diagram.

The group easily reached the consensus that seven of the potential causes were suspected of contributing to the field problem. Eric agreed to set up the experiment assuming two levels for each factor, and the others determined what those levels should be to relate the experiment to the production reality. Eric returned to the group and announced that he was able to use an L8 orthogonal array to set up the experiment and that eight tests were all that were needed at this time. The test matrix for the seven suspected factors is shown in Table 9.6.

Table 9.6: The Test Matrix for the Seven Factors
Test Number	Levels for Each Suspected Factor for Each of Eight Tests
Test Number	C1	C2	C7	C11	C13	C15	C16
1	1	1	1	1	1	1	1
2	1	1	1	2	2	2	2
3	1	2	2	1	1	2	2
4	1	2	2	2	2	1	1
5	2	1	2	1	2	1	2
6	2	1	2	2	1	2	1
7	2	2	1	1	2	2	1
8	2	2	1	2	1	1	2

Eric explained that this matrix would allow the group to determine if a difference in test responses existed for the two levels of each factor and would prioritize the within-factor differences. Since the two levels of each factor represented an actual situation that existed in production during the time the failed parts were produced, this information could be used to correct the problem. By now, Stephen had identified a test procedure and response that seemed to fit the requirements outlined in this section.

Two weeks were required to gather all the material and parts for the experiment and to run the experiment. The test results are shown in Table 9.7. While Eric entered the data into the computer for analysis, Timothy and Christine plotted the data to see if anything was readily apparent. The factor level plots are shown in Figure 9.4.

Table 9.7: Test Results
Test Number	Result
1	10
2	13
3	15
4	17
5	14
6	16
7	19
8	21

Figure 9.4: Plots of averages (higher responses are better).

When Eric finished with the computer, he reported that of all the variability observed in the data, 53.65% was due to the change in factor C2; 33.38% was due to the change in factor C1; and 11.92% was due to the change in factor C11. The remaining 1.04% was due to the other factors and experimental error. The large percentage variability contribution, coupled with the fact that the differences between the levels of the three factors are significant from an engineering standpoint, indicate that these three factors may indeed be the culprits. The computer analysis indicated that the best estimate for a test run at C1 = 2, C2 = 2, and C11 = 2 is 21.4. One of the eight tests in the experiment was run at this condition and the result was 21. Two confirmatory tests were run and the results were 11 and 20. The group then moved into a second phase of the investigation to identify what the specs limits should be on C1, C2, and C11. In the second round of testing, eight tests were required to investigate three levels for each of the three factors. The setup for the second round of testing involved an advanced procedure (idle column method) that will be presented later in this chapter, so the example will be concluded for now.

In summary, the group in the example took the following actions:

Gathered appropriate backup data
Called together the right experts
Made a list of the possible causes for the problem
Prioritized the possible causes
Determined the proper test procedure and response to be measured
Reached agreement prior to running any tests
Approached the investigation in a structured manner
Asked and addressed one question at a time

Obviously, there are many ways to approach a particular DOE. In a situation where testing or material is very expensive, the most efficient experimental layout must be used. In the following sections, techniques are introduced that help the experimenter optimize the experimental design. Additional opportunities to optimize the experiment should be examined. Consider the situation where there is a five-part process. A brainstorming group has constructed a cause-and-effect diagram for a particular process problem. The number of suspected factors for each part of the process is shown in Figure 9.5.

Figure 9.5: A linear example of a process with several factors.

The obvious approach would be to set up the experiment with 21 factors. An alternative approach would be to consider only seven factors for the first round of testing. These would be the six factors within part 5 plus one factor for the best and worst input to part 5. If the difference in input to part 5 is significant, then the investigation is expanded upstream. The decision to approach a problem in this manner is dependent upon the beliefs of the experts. If the experts have a strong prior belief that a factor in part 1, for instance, is significant, then a different approach should be used. This approach is also dependent upon the structure of the situation.

The above example is presented to illustrate the point that the experimenter should be alert for ways to test more efficiently and effectively.

MISCELLANEOUS THOUGHTS

An additional useful method of looking at the data is to plot the contrasts on normal probability paper. For a two-level factor, the contrast is the average of all the tests run at one level subtracted from the average of the tests run at the other level. For the example in this section, the contrasts are shown in Table 9.8.

Table 9.8: An Example of Contrasts
Factor	Average at Level One	Average at Level Two	Contrast (Level 2 Avg. - Level 1 Avg.)
C1	13.75	17.50	3.75
C2	13.25	18.00	4.75
C7	15.18	15.50	-0.25
C11	14.50	16.75	2.25
C13	15.50	15.75	0.25
C15	15.50	15.75	0.25
C16	15.50	15.75	0.25

These contrasts are plotted on normal probability paper versus median ranks. The values for median ranks are available in many statistics and reliability books and are used in Weibull reliability plotting. For this example, the normal contrast plot is shown in Figure 9.6.

Figure 9.6: Contrasts shown in a graphical presentation.

To plot the contrasts on normal paper, the contrasts are ranked in numerical order, here from -0.25 (C7) to 4.75 (C2). The contrasts are then plotted against the median ranks or, in this case, against the rank number shown on the left margin of the plot. Factors that are significant have contrasts that are relatively far from zero and do not lie on a line roughly defined by the rest of the factors. These factors can lie off the line on the right side (level 2 higher) or on the left side (level 1 higher). In the example, two separate lines seem to be defined by the contrasts. This could be due to either of these situations:

C1, C2, and C11 are significant and the others are not.
There may be one or more bad data points that occur when C1, C2, and C11 are at one level and the other factors are set at the other level.

In this example, C1, C2, C11, and C16 were at level 2 and the other factors were set at level 1 for run number eight. Depending upon the situation, it would be worthwhile to either rerun that test or to investigate the circumstances that accompanied that the test (e.g., was the test hard to run because of the factor settings or did something else change that was not in the experiment?). In the example, this combination of factors represented the best observed outcome, and the confirmation runs supported the results of the original test.

Plotting contrasts is a way of better understanding the data. It helps the experimenter visualize what is happening with the data. Sometimes, information that might be lost in a table of data will be crystal clear on a plot.