What is the proper role of tests of statistical significance in social science research?
What are complementary methods to statistical significance testing for determining the replicability of results in research experiments?
INTRODUCTION
Introduction to the Logic and Process of Significance Testing:
1. Set up a null hypothesis and an alternative about the population or populations.
2. Set up an alpha level. An alpha level is the probability level you view as low enough to constitute evidence that there is a contradiction between the data and the assumption that the null hypotheis is true in the population (often alpha is set at .05 in the behavioral sciences).
3. Gather data from a sample.
4. Compute the value of a test statistic based on the sample data.
5. Compute the probability of the value of the test statistic in Step 4 under the assumption that the null is true (usually given in a table or as part of a print-out).
6. If the probability in step 5 is less than alpha selected in step 2, then conclude that there is an inconstancy between the null hypothesis and the data. You can then reject the null hypothesis in favor of the alternative hypothesis and state that the results are statistically significant.
If the probability is greater than the alpha level, then conclude that the sample data is consistent with the null hypothesis. You must then fail to reject the null hypothesis and state that the results are not statistically significant.
Note that statistical significance is not the same as practical significance.
For example, the null hypothesis is often something like
population_mean_1 = population_mean_2
Rejecting this null hypothesis only indicates that the sample data imply that there is some difference in the population; however, that difference may be small and unimportant.
[Table of Contents]
MAJOR BIBLIOGRAPHIC RESOURCES
[Check your local library or bookstore for access]
Chow, S.L. (1996). Statistical significance: rationale, validity and utility.
Thousand Oaks, CA: Sage.
Harlow, L.L., Mulaik, S.A., & Steiger, J.H. (Eds.). (1997). What if there
were no significance tests? (Multivariate Applications Book Series). Mahwah, NJ:
Lawrence Erlbaum Associates.
McLean, J.E. & Kaufman, A.S. (Eds.). (1998). Statistical significance testing
[Special Issue]. Research in the Schools, 5(2). Birmingham, AL: Mid-South
Educational Research Association.
Mohr, L.B. (1990). Understanding significance testing. (Quantitative
Applications in the Social Sciences No. 07-073). Newbury Park: Sage.
New ways in statistical methodology: from significance tests to Bayesian
inference. (1998). (European University Studies Series VI, Psychology). New York:
P. Lang.
[Table of Contents]
