Hypothesis Testing: Two Samples

Sampling Distribution of the difference between sample means:

By randomly drawing individual samples from a population and calculating their means, we were able to construct a frequency distribution, which we call the sampling distribution of the means.

There are several important properties of the sampling distribution of the means:

When we draw pairs of independent samples from the population, we construct a frequency distribution that we call the sampling distribution of the differences.

Independent Samples:

~Samples drawn according to random selection procedures so that the choice of one observation for a sample does not affect the probability of another observation being chosen for a different sample.

~Samples in which the behavior of the members of one sample is not related to the behavior of the members of another sample.

Several properties characterize the sampling distribution of the means:

We know we can compute t-scores (derived from the logic of z-scores) to evaluate these two means. This is the basis of the mean difference hypothesis test, and provides a statistical test of a hypothesis about the difference between two means.

We can calculate the t-scores using the formula:

The definitional formula for independent t-tests:

Assumptions behind the use of the t-test:

In order for the t-test to be valid in the determination of significance of the difference between two means, the following assumptions must be met:

Type I and Type II Errors:

As we undertake research, we are confronted with the fact that, based on the data, there is always the possibility of making the wrong decision about the significance of the data.

There are two types of error that we can make:

1. Type I or alpha errors.

These are errors that occur when we reject the null hypothesis when the null hypothesis is true. This is the same thing as chance or luck producing the results we find, even though the probability is low. Its probability is alpha.

Type II or beta errors.

These are the errors that occur when we fail to reject the null hypothesis when the null hypothesis is incorrect. In other words, there are significant differences between the sample mean and the population mean and our test did not detect those differences. Its probability is beta.

Based on the example of the test of the effectiveness of assertiveness training, we can consider:

The larger value we select for alpha, the more likely we are to reject the null. If we can't accept the null, we can't make a type II error.

The smaller value we select for alpha, the more likely we are to accept the null. In accepting the null, we risk making a type II error.

There are costs associated with making either type error!

If we make a type I error (in the example), we would spend scarce resources for an ineffective intervention.

If we make a type II error (in the example), clients would forgo the benefit of a needed and effective intervention.

Power:

Statistical power analysis deals with the probability of avoiding type II errors. In other words, to assesses the probability of correctly rejecting a null hypothesis that is false.

Power is the probability that a statistical test will detect a false null hypothesis, or detect a true difference when one is present.

Power is defined as 1 - beta.

There are various techniques that can be used to increase Power:

Reliability of the instrument. The more reliable the measures being analyzed, the greater the likelihood that a true difference will be determined to be significant. Reliable scores are dependable, consistent, and not overly prone to measurement error.

Hypothesis Testing: Two Samples

Dependent Samples:

A dependent sample occurs when the data from two samples drawn from a normal population are paired or matched. That is to say that the scores in one sample are effected by the scores on another.

Examples of dependent samples could include:

Within-subject comparisons - the same individual receives both treatment conditions (more common in medical science)

Repeated measures - where a subject's pre-test scores are matched with post-test scores, or a subject's score at time 1 is matched with scores at time 2, time 3, etc.

Due to the effects of measurement, each prior testing will effect the scores on the measurement.

This results in the data being correlated.

The estimated standard error of the mean differences is determined by the formula

This formula can be used if you know the correlation coefficient (r) between measurement periods.

Formula for t-test with related samples:

This can be a relatively difficult formula to use, so a more useful method has been developed, termed the direct difference method.

Direct Difference Method:

we obtain the computational formula for t-test for dependent samples:

where the mean difference is computed by the formula

Example:

Data is randomly collected from nine graduate students in an attitude survey about taking a statistics class, both before they've taken the statistics class and at the end of the semester they take the class. Since each student's post-test score could, at least in part, be determined by his/her pretest score, regardless of the class, the dependent t-test formula is chosen to test the null hypothesis.

Since it's possible the class could be detrimental to their attitude about statistics, as well as enhancing their attitude, a two-tailed test is chosen.

The hypothesis we are interested in are:

In the first case, the class would have had a strong positive influence on attitudes about statistics to be true. In the second, the class would have a negative influence on the subjects to be true. In other words, higher scores on the survey correspond with better attitudes about statistics.

Step 1. The null hypothesis is stated and tested as

Step 2. Extract the necessary data.

Pre and Post Class Attitudes

Step 3. Write out the formulae you will use.

Step 4. Follow the formulae exactly as written, placing the appropriate values into each formula and solve.

Step 5. Determine the appropriate degrees of freedom. In this case, there are nine pairs of data; df = N - 1, so there are 9 - 1 df, or 8 df.

Step 6. Look at the table in the text to determine the critical value with alpha = .05 and df = 8.

The critical value is 2.3060

Step 6. Make a decision.

Based on the fact that

the observed mean difference is not due to sampling error (chance); there is a difference between populations. Since the value is negative (ONLY in our HYPOTHETICAL EXAMPLE, of course), we would conclude that the class had a negative impact on the subjects attitudes.

The assumptions for the use of dependent sample t-test are the same as those used for the independent sample t-test.