EVALUATION PHASE

Evaluating the ideas developed during the creative phase is a critical step in the job plan. The ideas generated will include practical suggestions as well as wild ideas. Each and every idea must be evaluated without prejudice to determine if it can be used or what characteristics the idea has that may be useful.

Proper evaluation of the ideas is a critical step. Remember, if an idea is discarded without thorough evaluation, the key to a successful solution may be lost. The time to create ideas is in the creative phase. If an idea is discarded, there may not be another opportunity to develop it again.

Evaluation processes can range from the simple to the complex. The methods selected depend to some degree on the number and quality of the ideas generated. (It is not uncommon to have several hundred ideas to evaluate.) In the evaluation process, do not be too critical. Look for the good rather than the bad and do not present unnecessary roadblocks .

The initial screening will weed out worthless ideas and sometimes generate new ideas or variations of the present ones. The initial screening will also begin to classify the ideas into basic groups that, in effect, constitute a second stage in the screening process. After the initial screening, it may be necessary to resort to systems designed to aid the process. Two favored, because of their simplicity, are paired comparison and Pareto voting. When the initial list of ideas has been screened and evaluated and reduced to a choice between several alternatives, evaluate the good and bad features of each alternative. Watch out for roadblocks, and try to determine if they can be eliminated and how they may be eliminated.

Experience has shown that this evaluation process is a difficult task. The impulse to quickly screen through the list to zero in on the best ideas must be controlled. The mass of data must be handled systematically to obtain maximum benefit from the creative phase. Careful screening is essential to isolating the best concept to carry over into the planning phase where the idea will be developed into a practical recommendation for action.

SELECTION AND SCREENING TECHNIQUES

A difficult problem that frequently confronts decision makers is the need to organize a large amount of data so that one or several of the most important items may be identified. It may be required to determine which of several alternatives appears to be the best, or it may be necessary to select a number of items so that they may be ranked and weighted by order of importance or some other criteria. Experience has shown that most people are not able to handle this task quickly and effectively. For this reason, it was decided to develop a simple method that would be applicable in most cases. More complex situations may require more sophisticated methods. However, experience has shown that a combination of two simple methods, Pareto voting and paired comparisons, will satisfy a majority of requirements.

Pareto Voting

Pareto voting is based on Pareto's law of maldistribution. Vilfredo Pareto (1846 “1923), a political economist , observed a common tendency of wealth and power to be unequally distributed. This observation has been refined to the degree that it can be said that there is an 80/20 percent relationship between similar elements. For example, twenty percent of the parts in an assembly contain eighty percent of the cost. This is most useful information in cost estimating; however, the relationship holds for many diverse examples such as the following:

• Twenty percent of the states use eighty percent of the fuel.

• Twenty percent of the activities create eighty percent of the budgeted expense.

• Twenty percent of the items sold generate eighty percent of the profit.

In value engineering, it is frequently necessary to select the best ideas, the highest value functions, the highest potential projects, or any of a number of other requirements. It has been found that the application of Pareto voting can help to simplify the list and will in most cases ensure that the most important items have been selected. It also produces results quickly and can be incorporated into the value engineering process to allow continuous operations without undue disruptions.

Pareto voting is conducted by requesting each team member to select what he or she believes are the items or elements that have the greatest effect on the system. This list of items is limited to twenty percent of the total number of items. For example, each team member would be allowed to select six items out of a list of 30. The vote is on an individual basis to obtain as much objectivity as possible.

The resultant lists are then compared and arranged into a new consolidated list in descending order by the number of votes each item received. Usually, several items will have been selected by two or more team members . The top 10 to 15 items are then ranked and weighted in a second step by using paired comparisons.

Paired Comparisons

Paired comparisons, or numerical evaluation as it is sometimes called, compares a list of items to rank and weight them in order of importance or some other criteria. Ranking is the assignment of a preferred order of importance to a list of items. Weighting is the determination of the relative degree of difference between items.

In paired comparisons, each item is compared to every other item on the list in turn , using a simple matrix. It is most convenient for up to 15 items; however, the limit is only for convenience. In most cases, ranking and weighting of long lists may be more practically done by direct magnitude estimation (DME).

A comparative decision is made between any two items on a two-level basis. There is either a great difference or a minor difference. The decision can be made based on the length of time it takes to decide. If there is no question as to which item to select, there is a great difference. If thought must be put into the decision, it would then be a minor difference. A major difference is weighted a 2, a minor difference a 1.

The paired comparison worksheet provides for the list to be evaluated and the evaluation grid. Start by transferring the list of items to the worksheet. Now compare A to B, A to C, etc. comparing A to each item of the list. A is then dropped and B compared to C, to D, etc. on through the list. B is then dropped and C is compared to each item on the list until every item has been compared to every other item. The following example will illustrate the process.

It is desired to select a vacation from among the following areas: Majorca, Florida, Colorado, or Greek island cruise. The first step is to list the locations on the evaluation summary area of the worksheet as shown in Table 12.3. The second step is to begin to compare the items.

Table 12.3: The Worksheet for Setting the List

Key Letter	Alternatives	Weight
A	Majorca
B	Florida
C	Colorado
D	Greek island cruise

Evaluation Summary

From the evaluation summary list, compare A to B, Majorca to Florida, and place the selected location letter in the A-B box of the evaluation grid. If the difference is major or clearly in favor of A, place a suffix 2 after the letter A. The A-B box should read A2. Now compare A to C. If the selection is A, place an A in the A-B box. If the difference is great, again add the suffix 2. Now compare A to C. If A is again the selection, place the A in the A-C box. If it requires thought to make the decision, the numerical suffix should be 1, minor. Drop the A and now compare B to C and B to D. Lastly, drop the B and compare C to D ” see Table 12.4.

Table 12.4: Evolution Summary

	B	C	D
A	A2	A2	A1
	B	C2	D2
		C	D1

To determine the ranking and weighting, add up the As, Bs, Cs, etc. In the example the result is as shown in Table 12.5.

Table 12.5: Ranking and Weighting

Key Letter	Alternatives	Weight
A	Majorca	5
B	Florida
C	Colorado	2
D	Greek island cruise	3

This analysis shows Majorca to be the most desirable. It is 40 percent more desirable than a Greek island cruise and 60 percent more desirable than Colorado.

Matrix Analysis

Although Pareto voting and paired comparison satisfy the screening and evaluation process in most cases, there are times when a more detailed analysis is required.

Two such cases could be when a decision involves large financial outlays or when serious consequences could result from a change. In these cases, every effort must be made to base a decision on the most objective data possible. For many of these decisions, there is a need to rank and weigh a number of alternatives against a series of specific criteria. By doing this, we learn which trade-offs must be made for the various requirements of the project, enabling us to make the best decision. In these cases, a combinex method is recommended.

Combinex was developed by Fallon (1971) and is based on comparing a number of alternatives to a series of criteria. Each alternative is compared to the criteria in turn and given a specific numerical rating. The resultant analysis clearly ranks and weighs each alternative against each criterion, which allows for trade-offs based on clearly defined data. This makes it an excellent tool in decision making.

Example

To illustrate the process, a typical problem familiar to most people will be used. The problem is to select an automobile for purchase. The criteria for selection have been taken from a list of factors affecting the sale of most products. The criteria selected will have a different value for each individual and have been chosen to illustrate several points. The selection criteria are:

1. Styling

2. Comfort

3. Reliability

4. Selection (models available)

5. Image

6. Cost

7. Economy (mi/gal)

In other instances, the criteria used could be the factors affecting the purchase of manufacturing equipment, location of a plant, construction of various types of facilities, or any other requirement involving a series of criteria for selection.

The alternatives to be considered for purchase are the XXXX models listed below along with their fictitious base prices. The analysis was made in April XXXX. The same analysis made in September XXXX might have resulted in a different conclusion as time and opinions change.

Alternatives
Ford	$ 14,000
Plymouth	$ 13,600
Chevrolet	$ 14,500
Honda	$ 15,000
Audi	$ 28,000

Rank and Weigh Criteria

The first step in the process is to decide the importance of the various criteria since each does not have an equal weight or bearing on the selection. In other words, the selection criteria must be ranked and weighed. To do this we will use the method of paired comparisons. A team of five persons applied paired comparisons as seen in Table 12.6.

Table 12.6: Criteria Affecting Car Purchase XXXX ” Paired Comparison

	B	C	D	E	F	G	Coding & Results
A	A1	A1	D1	E1	F1	A1	F - Cost 6
	B	B1	B1	E1	F1	G1	G - Economy 4
		C	C1	E1	F1	G1	E - Image 4
			D	E1	F1	G1	A - Styling 3
				E1	F1	G1	B - Comfort 2
					F	F1	C - Reliability 1
							D - Selection 1

The result of the group's analysis is their opinion. Another group would apply their own values and probably produce a different result. This group 's ranking and weighing shows cost to be the most important criterion. Cost was six times more important than comfort and 50 percent more important than economy.

Evaluate Each Alternative

The criteria values are entered into the combinex scoreboard as illustrated in Table 12.7.

Table 12.7: Criteria Weighing

Criteria
	A	B	C	D	E	F	G	Total	Rank
Weight	3	2	1	1	4	6	4
Alternatives
Ford
Chrysler
Chevy
Honda
Audi

Next, the team compares each alternative, in turn, to each of the criteria. A value is then placed in the upper section of its respective box. These values are based on the criteria weighing scale shown below.

5	Superior
4	Good
3	Average
2	Fair
1	Poor

In this example, the comparison was made as shown in Table 12.8.

Table 12.8: Criteria Comparison

Criteria
	A	B	C	D	E	F	G	Total	Rank
Weight	3	2	1	1	4	6	4
Altematives
Ford	2/	4/	3/	4/	3/	3/	4/
Chrysler	4/	4/	4/	5/	3/	3/	4/
Chevy	4/	4/	3/	4/	3/	3/	4/
Honda	4/	3/	3/	3/	3/	3/	5/
Audi	3/	4/	3/	3/	4/	1/	3/

How does the Ford satisfy the styling criteria in the opinion of the selection team? The team decided it was fair and rated it a 2. For reliability, the team said the Ford was average and weighed it a 3. After the Ford was compared to each criterion in turn, the second alternative, the Chrysler, was compared. In each case, each team member expressed an opinion individually. In some instances, it was necessary to develop an average. In other cases, the decision was unanimous. This was done until each alternative was compared to each criterion.

The third step of the process is to multiply the criteria weight by the comparison value as shown in Table 12.9. For example, the Ford styling weight of 3 was multiplied by the value of 2. The resultant product of 6 is inserted in the lower section of the box. After completion of each individual weighing, the score is summed under the total column.

Table 12.9: Criteria Weight Comparison ” Completed Matrix

Criteria
	A	B	C	D	E	F	G	Total	Rank
Weight	3	2	1	1	4	6	4
Alternatives
Ford	2/6	4/8	3/3	4/4	3/12	3/18	4/16	67	4
Chrysler	4/12	4/8	4/4	5/5	3/12	3/18	4/16	75	1
Chevy	4/12	4/8	3/3	4/4	3/12	3/18	4/16	73	3
Honda	4/12	3/6	3/3	3/3	3/12	3/18	5/20	74	2
Audi	3/9	4/8	3/3	3/3	4/16	1/6	3/12	57	5

The total score is shown in the column at the right, and the choices are ranked in the far right column. This analysis shows the first choice to be the Chrysler and the last choice to be the Audi, as illustrated in the complete combinex scoreboard (Table 12.11).

Analyze Results

An analysis of the table shows that although the Audi was a poor fifth in the selection process, the primary reason was cost. If the cost had been average, the additional 12 points would have raised Audi's total above that of the Ford. The table also shows that if the Ford styling had been rated as good, 4, this car would have been ranked second with a score of 73. Although styling was originally ranked fourth in importance with a 3, other factors may now be considered. An improvement in reliability would not have a major effect on the overall rating, but a reduction in cost or an improvement in economy could have. Cost could be negotiated; economy would require some basic product changes.