IS THE DIFFERENCE REAL?

The scale on the distribution is marked with the actual values and with "standardized" values, which are computed by dividing the differences by the standard error. Looking at standardized distances is convenient , since the percentage of cases within a standardized distance from the mean is always the same. For example, 34% of all samples are between zero and one standardized unit greater than the mean, and another 34% are between zero and one standardized unit less than the mean. If you always express your distances in standardized units, you can use the same normal distribution for evaluating the likelihood of a particular difference.

From Figure 5.1, you can see that about 13% of the time, you would expect to have at least a 7-point difference in the sample means when two population means are equal. Why? You just found out that the 7-point difference is 1.5 standard errors. Look on the figure to see what percentage of the differences is that big. You should look at the area to the right of +1.5 and the area to the left of -1.5. Since the distribution is symmetric, the two areas are equal. Each one is about 6.7% of the total, and together they make up a little over 13% of the total. So about 13% of differences in means are going to be as big as 1.5 standard errors (or 7 points) if the real difference in means is zero.

WHY THE ENTIRE AREA?

You may wonder why you do not find the probability of getting a difference of just seven. Think of the following analogy. You are tired of the life of a poor student and have decided that the quickest (legal) way to upgrade your status is to marry rich. You settle on a definition of rich. Perhaps you need an income of $250,000 a year. Now you want to see how likely it is that you can achieve your goal. You go to the university library and ask the reference librarian to find some facts. Would you ask about just the number of eligible singles of the opposite sex with incomes of $250,000? No, you would ask for incomes of $250,000 or more, since they all satisfy your criterion of richness. In evaluating your chances of marrying rich, you would include all incomes of $250,000 or more. Similarly, when you are trying to decide whether seven is a likely outcome for a difference, your interest is not just in the number seven but in all differences that are at least that large.

Since both outcomes were possible and you did not know the outcome before you actually ran your experiment, when you evaluate the chances of seeing a difference at least as large as seven points, you have to look in both directions. Both of the extreme regions of the distribution are atypical. Sometimes, though, you can look in just one direction. It really depends on how you stated your initial hypothesis. If you hypothesized that you expect higher values you would be looking at the right tail, if lower you would be looking at the left tail of the distribution. This type of test is called a one-tailed test . Your decision to use a one-tailed test is a very important one because once you see the results you cannot go back and switch sides and apply the one-tailed test for a difference that is in the other direction. Use a one-tailed test only if you definitely expect one specific group to be higher. Otherwise, use a two-tailed test and look at both sides of the distribution.