So now the question is, "Is the difference real?" Usually when you conduct a study, you have some ideas that you want to explore. These ideas, often called hypotheses, typically involve comparisons of several groups, such as "Do men and women find life equally exciting?" "Does income
Chances are if you compare two or more things, you are going to find some differences. After all, no two things are exactly the same. The question is not so much if things are different but rather, what can you make of the difference?
So far we have seen that different samples from the same population give different results. The real issue is, how much will they differ? How can you decide whether a difference in sample means can be attributed to their natural variability or to a real difference between groups in the population?
How can you decide when a difference between two means is big enough for you to believe that the two samples are from a population with different means? It depends on how willing you are to be wrong. Look at Figure 5.1, which is the real distribution of differences for samples of
Figure 5.1: Theoretical distribution of differences of means.
The scale on the distribution is
From Figure 5.1, you can see that about 13% of the time, you would expect to have at least a 7-point difference in the sample means when two population means are equal. Why? You just found out that the 7-point difference is 1.5 standard errors. Look on the figure to see what percentage of the differences is that big. You should look at the area to the right of +1.5 and the area to the left of -1.5. Since the distribution is symmetric, the two areas are equal. Each one is about 6.7% of the total, and together they make up a little over 13% of the total. So about 13% of differences in means are going to be as big as 1.5 standard errors (or 7 points) if the real difference in means is zero.