MULTIPLE COMPARISON PROCEDURES

A significant F value tells you only that the population means are probably not all equal. It does not tell you which pairs of groups appear to have different means. You can reject the null hypothesis that all means are equal in several different situations. For example, people who find the ride comfortable may differ in age from those who find the ride rough but not from those who find the ride normal. Or people who find the ride comfortable may differ in age from both of the other groups. In most situations, you want to pinpoint exactly where the differences are. To do this, you must use multiple comparison procedures.

Why do you need yet another statistical technique? Why not just calculate t tests for all possible pairs of means? The reason for not using many t tests is that when you make a lot of comparisons involving the same means, the probability that one out of the bunch will turn out to be statistically significant increases . For example, if you have five groups and you compare all pairs of means, you are making 10 comparisons. When the null hypothesis is true (that is, all of the means are equal in the population), the probability that at least one of the 10 observed significance levels will be less than .05 is about .29. If you keep looking, even unlikely events will happen. The more comparisons you make, the more likely it is that you will find one or more pairs to be statistically different, even if all means are equal in the population.

Multiple comparison procedures protect you from calling too many differences significant. They adjust for the number of comparisons you are making. The more comparisons you make, the larger the difference between pairs of means must be for a multiple comparison procedure to report a significant difference. So, you can get different results from multiple t tests than from multiple comparison procedures. Differences that the t tests find significant may not be significant based on multiple comparison procedures. When you use a multiple comparison procedure, you can be more confident that you are finding true differences.

Several different procedures can be used to make multiple comparisons. The procedures differ in how they adjust the observed significance level for the fact that many comparisons are being made. Some require larger differences between pairs of means than others. For further discussion of multiple comparisons, see Kirk (1968).