The Concept of Statistical Significance Testing
A frequently asked question of:
The Clearinghouse on Assessment and Evaluation [ERIC/AE] 1129 Shriver Laboratory University of Maryland - College Park College Park MD 20742 Toll Free: 1.800.464.3742 E-mail: askae#064;ericae.net WWW: http://ericae.net
As a postsecondary student who is studying inferential statistics, I need documentation to help me to understand the controversy that surrounds statistical significance testing.
What is the proper role of tests of statistical significance in social science research?
What are complementary methods to statistical significance testing for determining the replicability of results in research experiments?
Table of Contents
1. Set up a null hypothesis and an alternative about the population or populations.
2. Set up an alpha level. An alpha level is the probability level you view as low enough to constitute evidence that there is a contradiction between the data and the assumption that the null hypotheis is true in the population (often alpha is set at .05 in the behavioral sciences).
3. Gather data from a sample.
4. Compute the value of a test statistic based on the sample data.
5. Compute the probability of the value of the test statistic in Step 4 under the assumption that the null is true (usually given in a table or as part of a print-out).
6. If the probability in step 5 is less than alpha selected in step 2, then conclude that there is an inconstancy between the null hypothesis and the data. You can then reject the null hypothesis in favor of the alternative hypothesis and state that the results are statistically significant.
If the probability is greater than the alpha level, then conclude that the sample data is consistent with the null hypothesis. You must then fail to reject the null hypothesis and state that the results are not statistically significant.
Note that statistical significance is not the same as practical significance.
For example, the null hypothesis is often something like
Rejecting this null hypothesis only indicates that the sample data imply that there is some difference in the population; however, that difference may be small and unimportant.
©1999-2012 Clearinghouse on Assessment and Evaluation. All rights reserved. Your privacy is guaranteed at