WHY IS THAT SO COMPLICATED?

You are probably wondering why all of a sudden we use verbiage that perhaps is not very scientific. Why are we going around in circles? Why do we have to assume that no difference exists between the means in the population and then figure out how likely the observed results are if no difference exists? Why not just calculate the probability that a difference exists? That is what we really want to know, is it not?

Although it sounds like a good idea, in this situation you cannot calculate the probability that a difference is present. A difference either exists or it does not. If we have two sample means, say 11 and 12, what do they tell us about whether it is true that the two means in the population are different? Not much. The probability of getting two sample means that differ by at least one depends on how much of a difference is present in the population. It depends on whether the true difference is 0, 1, 2, 4, 100, or whatever. A difference of one may be very unlikely if the true difference is 100 but perfectly likely if the true difference is zero or two. But we do not know what the true difference is. We can only consider the likelihood of a value of one or more in relation to some hypothetical situation, such as a true difference of zero or a true difference of 100. We cannot assign the difference some overall probability.

What if you found that the two sample means were 11 and 11? Would you claim that it is certain the two means are exactly equal in the population? Would you be willing to forget the possibility that the population means might be 11 and 11.1? Of course not. (I hope). You have seen that samples vary and that it is most unlikely for two sample means to be exactly equal even if the two means are equal in the population. Similarly, you can easily get sample means that are the same from populations whose means differ to a small extent.

You simply cannot figure out the probability that two population means are equal or unequal . You can, however, estimate the probability that you would see a difference of at least two (or some other value) in the sample when no difference exists in the population (or when a difference of a particular size is present). In the previous example, you saw the calculations for the probability that the means from two samples would differ by at least seven when no difference exists in the population.

To test a hypothesis, you do the following:

1. State the hypothesis of interest. This is what you think is really true for the population.

2. Determine the frame of reference you will use to evaluate your hypothesis. This is what is true in the population if your hypothesis is wrong. This "frame of reference" is called the null hypothesis , since it describes the population when the hypothesis you are interested in is not true, when it is null.

3. Calculate the probability that you would see a difference at least as large as the one you observed in your sample if the null hypothesis is true.

4. If this probability (called the observed significance level) is small, say less than .05, reject the null hypothesis.

5. If the observed significance level is large, do not reject the null hypothesis. This does not mean that you accept the null hypothesis. You simply do not reject it. You remain uncertain .

You must state the null hypothesis in a way that allows you to calculate the distribution of sample means when it is true. You cannot use a null hypothesis that says the population means are unequal, since no single distribution of sample means exists for that statement. But you can have a null hypothesis that says the difference between two population means is five or some other particular number. The null hypothesis must provide the reference point for calculating the probability of the observed results. You calculate the probability of the observed results if the null hypothesis is true.