SIGNIFICANCE VS. IMPORTANCE

What does it mean if you reject the null hypothesis that two population means are equal? Does it mean that an important difference exists between the two groups? Not necessarily . Whether a difference of half a year of education between two groups is found to be statistically significant or not depends on several factors. It depends on the variability in the two groups, and it depends on the sample sizes.

A difference can be statistically significant with a sample size of 100, while the same difference would not be significant with a sample size of 50. The difference between the two sample means is the same, half a year, but its statistical interpretation differs . For large sample sizes, small differences between groups may be statistically significant, while for small sample sizes, even large differences may not be.

What can you make of this? Finding that a difference is statistically significant does not mean that the difference is large, nor does it mean that the difference is important from a research point of view. For sufficiently large sample sizes, you might find that even a small difference is statistically significant. That does not mean that the difference is of any practical importance. For example, we all know that on the average, an extra month of education will not do much for you. It is unlikely to alter your perception of the world. It probably does little to enhance your ability to explain why some people find life exciting and others do not.

On the other hand, if the sample sizes in the two groups are small, even a difference of four years of education may not appear to be statistically significant. In this case, you do not want to rule out a variable that may prove to be an important variable. Instead, you must worry about the fact that with small sample sizes you can miss important differences. You must allow yourself the possibility that a big difference exists because your ability to find it is poor. The probability of detecting a difference of a particular magnitude when it exists is called the power of a test. You can estimate in advance how big a sample you need in order to detect a difference that you consider really important. Discussion of how that is done is a little beyond this book. However, the reader should understand that the power is actually determined by three factors:

• Effect size ” The probability of achieving statistical significance is based not only on statistical considerations but also on the actual magnitude of the effect of interest.

• Alpha ( ± ) ” As alpha (Type 1 error) becomes more restrictive , power decreases. This means that as the experimenter reduces the chance of finding an incorrect significant effect, the probability of correctly finding an effect also decreases.

• Sample size ” At any given alpha level, increased sample sizes always produce greater power of the statistical test. But increasing sample size can also produce "too much" power. This means that when you increase sample size, smaller and smaller effects will be found to be statistically significant, until at very large sample sizes almost any effect is significant. Remember that small sample sizes will make the statistical test insensitive and that large samples will make it overly sensitive. Figure 6.1 shows an example of the impact of sample size on power for various alpha levels (.01, .05, .10). Needless to say, the experimenter must consider the impact of power before selecting the alpha level.

  
  Figure 6.1: Impact of sample size on power for various alpha levels (.01, .05, .10).

In summary, remember that even though two groups are found to be statistically different, their difference is not necessarily of practical importance. Evaluate the difference on its own merits.

How can you test the null hypothesis that two percentages are equal in the population? How can you test the null hypothesis that two variables are independent? Here are the things that you need to know and will be discussed later:

• Observed frequencies are simply the numbers of cases with specific combinations of values.

• Expected frequencies are the numbers of cases that would have specific combinations of values if the null hypothesis were true.

• The chi-square statistic is based on a comparison of observed frequencies with expected frequencies. From it, you can obtain an observed significance level for the hypothesis that two proportions are equal.

• Two variables are independent if knowing the value of one variable tells you nothing about the value of the other.

• The degrees of freedom of a cross-tabulation reflect the number of cells in the table that are free to vary. You compute them by taking the number of rows minus one and multiplying that by the number of columns minus one.

• From the chi-square statistic and the degrees of freedom in a cross-tabulation, you can calculate the observed significance level for the null hypothesis that the two variables are unrelated.

• Chi-square increases in direct proportion to sample size if the strength of the relationship stays the same. If you double the number of cases in each cell of a cross-tabulation, chi-square is doubled .