Thomas E. Dawson
Texas A&M University, January 1997
Counseling psychology is a burgeoning field still in its inchoate stages, attempting to gain/maintain autonomy and respect. As students of a scientific-practicing counseling psychology program, it behooves us to conduct well thought-out, meaningful research in the name of practicing "good science," as does it benefit all counseling psychologists in the name of furthering the field's namesake. Unfortunately, many times the tendency to embark on a research endeavor lacks the necessary foresight in constructing the design. Research designs are pervious to many different types of threats to their internal and external validity. In the traditions of Campbell and Stanley, and Cook and Campbell, this paper will elucidate some of the more common types of research designs, along with the coexistent threats to validity. Further, an example of each type of design has been given from the counseling literature for the interested reader to paruse and help make the concepts concrete.Paper presented at the annual meeting of the Southwest Educational Research Association, Austin, January, 1997.
Poem by Skyler Huck, University of Tennessee
Upon its inception as an experimental science, psychology has utilized the scientific method found in the physical sciences. In 1879 Wilhelm Wundt opened the first psychological laboratory in Leipzig, Germany, and with that commencement also came the first arguments about the validity of Wundt's experiments (Benjamin, 1988). Since that time, the scientific method has been applied to various psychological constructs, e.g., behaviorism, learning theory, Gestalt psychology, animal experimentation, cognition, and functionalism (Gannon, 1991).
Counseling Psychology has experienced many "growing pains" in its attempts at being recognized as a separate (Hiebert, Simpson, & Uhlemann (1992); see also Wooley, Duffy, & Dawson, (1996) for a preliminary study to support counseling psychology's attempts at autonomy), viable science, which has been linked previously by some to the utilization of the scientific method (Hill & Corbett, 1993; Schultz, 1972). Since the inauguration of counseling psychology in 1946 (Whitley, 1984) it was mainly an applied psychology. At this juncture in time, the application of psychology was only beginning to gain respect from the intact group of psychologists who considered themselves as "pure"--that is they engaged in experimental psychology (Benjamin, 1988). In light of this zeitgeist and the identity struggles within the division, it stands to reason that counseling psychology places scientific inquiry through the rigor of the scientific method, as a core function.
In the past 20 years, there has been growing dissension in the ranks of counseling psychology researchers regarding the way in which research focusing on the philosophy of science and counseling psychology is being conducted. Many believe that researchers are placing too much emphasis on objectives and questions to which research should be directed, with little attention to actual research designs and methods (Ford, 1984; Goldman, 1976; Howard, 1982; Parker, 1993; Serlin, 1987; Serlin & Lapsley, 1985). Others have directly stated that more attention should be placed on the training aspects of research methodology (Gelso, 1979a, 1979b; Goldfried, 1984; Magoon & Holland, 1984), (note though that Birk and Brooks, 1986, report that 81% of 300 counseling psychologists surveyed reported adequate training in research). Indeed, the results of a national survey of counseling psychologists indicate that 49.6% of their time is devoted to the activity of research (Watkins, Lopez, Campbell, & Himmell, 1986), thus further supporting the relevance of the present paper to counseling psychologists.
If this is the case, how is it that so many counseling psychologists and counseling psychology students are producing "bad science"? For example, in a specific instance of reviewing one (psychology-type) department's recent dissertations, Thompson (1994) found numerous blatant mistakes due to methodological errors, and many others have challenged the current state of research (e.g., O'Hear & MacDonald, 1995). These errors are most likely representative and indicative of the more common mistakes found in research presently.
With this backdrop and apparent need for remediation, the present paper presents a "primer on experimental designs," with the specific goal of (review), and the more comprehensive intention that through "better and improved science", counseling psychology will continue to solidify its place as an indisputably separate and viable field. In the traditions of Campbell and Stanley (1963) and Cook and Campbell (1979), a review of experimental and quasi-experimental designs and how threats to validity impacts counseling research will be presented, employing examples from the current counseling literature.
The Validity of Experimental Designs
Internal validity is one important type of research validity. The term "internal validity" refers to the extent that extraneous variables (error variance) in an experiment are accounted for. It is paramount to the researcher that model specification error variance (as distinct from measurement and sampling error variance) is controlled because if not, the researcher can not emphatically conclude that the observed outcome is due to the independent variable(s) (Parker, 1993). Campbell and Stanley (1963) stated that "internal validity is the basic minimum without which any experiment is uninterpretable" (p. 5). There are eight major threats to internal validity: (a) history, encompassing the environmental events occurring between the first and second observations in addition to the independent variable(s); (b) maturation, which refers to the processes within the participants (psychological and/or biological) taking place as a function of the passage of time, not attributable to the independent variable(s); (c) testing, which is sensitization to the posttest as a result of having completed the pretest; (d) instrumentation, which refers to deterioration or changes in the accuracy of instruments, devices or observers used to measure the dependent (outcome) variable; (e) statistical regression, which operates when groups are selected on the basis of their extreme scores, because these anomalous scores tend to regress toward the mean on repeated testing; (f) selection, which refers to the factors involved in placing certain participants in certain groups (e.g., treatment versus control), based on preferences; (g) mortality, which refers to the loss of participants and their data due to various reasons, e.g., death or sickness; and (h) interactions of previous threats with selection. For example, a selection-maturation interaction results when the experimental groups are maturing at different rates based on the selection of the participants (Campbell & Stanley, 1963). In later writings, Cook and Campbell (1979) identify an additional threat to internal validity. This is ambiguity about the direction of casual influence when all other plausible third-variable explanations have been ruled out of the A-B relationship, but it remains unclear as to whether A causes B, or B causes A.
This construct asks the question of generalizability. Which populations, settings, treatment variables and measurement variables can these results be generalized to? Generalizing across persons requires research samples to be representative of the population of interest. Generalizing across times and settings usually necessitates systematically administering the experimental procedure at different times and different settings (Parker, 1993). The inability to obtain samples that are representative of the populations from which they came, especially if studied in various settings, and at different times, results in the inability to generalize beyond the persons, time, and setting of the original study. Tests that do meet the representativeness criteria are, in essence, tests of statistical interaction. For example, if there is an interaction between a therapeutic treatment and ethnicity, then it can not be decisively stated that the treatment holds true across different ethnicities. When effects of differing magnitude exist, the researcher must delineate when and where the effect holds, and when and where it does not (Cook & Campbell, 1979).
The statistical interaction threats to external validity outlined by Cook and Campbell (1979) are as follows: Interaction of selection and treatment (as in the previous example dealing with ethnicity); and interaction of setting and treatment (e.g., can a casual relationship obtained on a military installation also be obtained on a university campus?). The last interaction is between history and treatment. In this case, the question involves to which period of the past or future can the results obtained be generalized. For example, the majority of experiments take place on university campuses, with undergraduate university students as participants. If an experiment was conducted on the day after a football loss to this university's arch rival, then the results may not generalize even to a week after the loss, much less beyond the participants and setting represented in the original study.
Parker (1993) reviewed and synthesized the Campbell and Stanley (1963) and the Cook and Campbell (1979) work and explicated two additional threats to external validity: The interaction of treatments with treatments, which refers to the administration of multiple treatments administered to the same participants, e.g., time-series designs wherein the effects may be cumulative; and the interaction of testing with treatment, not to be confused with the internal validity threat of testing when the pretest sensitizes the participant to the posttest. In the external validity case, the pretest may increase or decrease the participants responsiveness or sensitivity to the treatment.
The above description of the most common threats to internal and external validity lays the groundwork for the planning of research projects. With these potential pitfalls in mind, the researcher is now ready to begin to plan which treatment design will be implemented (Lysynchuk, Pressley, d'Ailly, Smith, & Cake, 1989). The following explanation of the different types of treatment designs and the inherent threats to their validity will use an "X" to represent the exposure of a group to an experimental treatment or event. An "O" will signify some type of observation or measurement. The Xs and Os in the same row will refer to the same group, and the order of the characters from left to right will designate the temporal order of the events. "R" will exemplify random assignment, if necessary (Campbell & Stanley, 1963).
Three Pre-Experimental Designs
In a review of the designs of the process and outcome studies published in the Journal of Counseling Psychology (JCP) between the years of 1964 through 1968, Kelley, Smits, Leventhal, and Rhodes (1970) found that 54% of the studies utilized a preexperimental design. Preexperimental designs are those in which there is no control group and/or have comparison groups that are formed nonrandomly, therefore yielding results which are difficult to interpret (Huck & Cormier, 1996). The three preexperimental designs presented by Campbell and Stanley (1963) are the one-shot case study, the one-group pretest-posttest design, and the static group comparison. We will examine these designs in the order given.
The one-shot case study
Much past research applied a design in which a single group was studied only once after a treatment was applied. These studies are diagrammed as follows:
According to Kelley Smits, Leventhal, and Rhodes (1970) the preponderance of designs they reviewed in the JCP were one-shot case studies (31%). Campbell and Stanley (1963) refer to these studies as having "...such a total absence of control as to be of almost no scientific value" (p. 5). They go on to state that "securing scientific evidence involves making at least one comparison" (p. 6) ...and that "It seems well-nigh unethical at least at the present time to allow, as theses or dissertations in education, case studies of this nature" (p. 7). As these studies are practically unused today, we will examine the threats inherent in the one-shot case design below, when they are associated with other more commonly used designs.
The one-group pretest-posttest design
This design is judged to be better than design one (Campbell & Stanley, 1976) and is a catalyst for understanding how many of the extraneous variables that threaten internal validity play out. The one-group pretest-posttest design can be reviewed by referencing Jemmett and Jemmett (1992), and is diagrammed as follows:
O1 X O2
In this design, history is one of the uncontrolled rival hypotheses, as the changes between O1 and O2 may have been due to events that possibly occured in addition to the experimenter's X. The longer the time that elapses between the two observations, and the more participants for which specific events happen collectively, then the more plausible history becomes as a rival hypothesis (Campbell & Stanley, 1963).
Other rival hypotheses include the participants maturing (physically or psychologically) between the pre and posttests, or possibly the participants do better on the posttest as a result of taking the pretest (testing). Maybe the measuring instrument changed over the course of the study (instrumentation), or certain participants may have selectively dropped out of the study (mortality/selection). If the participants scored atypically on the pretest, they may have regressed toward the mean naturally (statistical regression), without any influence of X (Huck & Cormier, 1996).
The static-group comparison
The third preexperimental design is the static-group comparison (e.g., Laser, 1984). In this design, a posttest is administered to two groups, one having been administered the X, and the other not (a control group). When diagrammed, this design appears as follows:
The basic problem with this design is the unknown status of the two groups prior to the administration of X, since the participants are not randomly assigned to the two groups. If a difference is obtained at posttest, these results may have been the influence of X. Alternatively though, the difference could have been an initial difference between the two groups. Since the participants either self-select themselves for participation into either group, or two existing groups are used, this is a selection threat to internal validity. Another threat to internal validity in this design that might threaten, even if the groups began as equal, is the selective drop-out rate of the participants in one group (mortality) (Campbell & Stanley, 1963; Huck & Cormier, 1996).
Three True Experimental Designs
True experimental designs yield results that are more trustworthy than the preexperimental designs due to the fact that random assignment is utilized, therefore reducing the amount of potential threats to internal validity (Huck & Cormier, 1996).
Pretest-posttest control group design
For an example of this design in practice, see Hains and Szyjakowski (1990) or Kush and Cochran (1993). The design is diagrammed as follows:
R O1 X O2
R O3 O4
Random assignment is employed to both groups, and both are given a pretest. One group is administered the X, and the other is not. A comparison of O2 and O4 should elucidate any effect of the X. The unique strength of this design is the addition of the pretest, though there are controversies surrounding the use of pretests after random assignment (Heppner, Kivligham, & Wampold, 1992). In this design, many of the previous internal threats to validity discussed so far are accounted for. The differences attributed to history (and hopefully instrumentation) between O1 and O2 would be similar to the differences between O3 and O4. The threat of maturation and testing should be equally manifested between the experimental and the control group, statistical regression, mortality, and selection interaction threats are protected by the random assignment of participants (Campbell & Stanley, 1963), occurring probably equally across the two groups.
Ironically, the major weakness of this design is in fact, its major strength, but for external validity reasons. The pretest would sensitize both the control group and the experimental group to the posttest in a like manner, therefore presenting no internal threat to validity. However, generalizing the results of a treatment that included a pretest in the design, to a different sample without a pretest, may yield much different results (Heppner et al., 1992).
The posttest-only control group design
Here, randomization is utilized to ensure the equalization of the two groups, without a pretest, as in the Van Noord and Kagan (1976) study. The design is depicted in this way:
R X O1
Again, random assignment is employed to both groups. The X is administered to the experimental group, and the second group acts as a control. The internal validity of this design is basically solid. According to Cook and Campbell (1979), the posttest-only control group design is the prototypical experimental design, and most closely exemplifies a condition in which a casual relationship can be discerned between an independent and dependent variable.
The main weakness of this design concerns external validity, i.e., the interaction of selection and treatment (Campbell & Stanley, 1963). Because of random assignment, the selection of subjects is not supposed to present a threat to internal validity. Nonetheless, it is often unknown whether the results of the study would generalize to another population (Heppner et al., 1992). For example, there could potentially be great differences between the results of a course on speed reading taught to a graduate class, versus a speed reading course taught to a high school class. Another problem deals with the absence of a pretest employed to reduce variability in the dependent variable. Random assignment is thought by some to account for this preexisting variability. But according to Huck and Cormier (1996) random assignment is not always random because (a) many researchers have a very loose definition of what randomization is, and (b) true randomization carries with it very stringent criteria and many researchers are unaware of the necessary precision and falsely believe they have true randomization, when they do not. The suggestion given by Huck and Cormier is that researchers explain definitively just how they accomplished randomization.
The Solomon four-group design
When a pretest is desired, but there is concern over the effects of using a pretest, as in the Bateman, Sakano, and Fujita (1992) study, this design is used, notationally described as:
R O1 X O2
R O3 O4
R X O5
This design is a combination of the pretest-posttest control group design (the first two groups), and the posttest-only control group (the last two groups). The main purpose of this design is to account for potential effects of the pretest on the posttest, and lends some degree of future replicability. For example, the Solomon four-group design accounts for the problem that the pretest-posttest control group design has, by comparing O2 to O5 to account for pretest sensitization, the only difference being that O2 receives a pretest prior to treatment. With regard to generalizability, the researcher can compare O2 to O4 and O5 to O6. If treatment effects are found in both cases, the results will be considered strong, and suggest future replicability, as one replication is confirmed with the data in hand (Heppner et al., 1992). The major drawback of this design is the amount of time, energy, and resources necessary to complete the study.
Three Quasi-Experimental Designs
When a true experimental design is not available to a researcher for various reasons, e.g., in clinical settings where intact groups are already formed, when treatment can not be withheld from a group, or when no appropriate control or comparison groups are available, the researcher can use a quasi-experimental design. As in the case of the true experimental design, quasi-experiments involve the manipulation of one or more independent variables and the measurement of a dependent variable. The major difference between true and quasi-experimental designs is the random assignment of participants (Heppner et al., 1992). Therefore, the internal validity of the quasi-experimental design is higher than that of the pre-experimental design, but lower than the true experimental design (Huck & Cormier, 1996). There are three major categories of quasi-experimental design: the nonequivalent-groups designs, cohort designs, and time-series designs (Cook & Campbell, 1979). An example of each category will be given, though the reader should be aware that there are many variations of each of the following examples.
The nonequivalent-groups design
The nonequivalent-groups design is the most frequently used quasi-experimental design (Heppner et al., 1992; Huck & Cormier, 1996). This design is similar to the pretest-posttest control group experimental design considered earlier. The difference is the nonrandom assignment of subjects to their respective groups in the quasi-experimental design. The design is diagrammed as follows, and can be further perused by referencing Braaten (1989):
Non R O1 X O2
Non R O3 O4
This design is one of most widely used designs in the social sciences because it is often interpretable. Cook and Campbell (1979) recommend this design when nothing better is available. The nonequivalent-groups design accounts for many of the threats to internal validity, except for four. The first uncontrolled threat is that of selection-maturation. As stated earlier, many researchers falsely believe that the administration of a pretest remedies the nonrandom assignment of participants, and use ANCOVA to "level" the groups. As has been succinctly pointed out by Loftin and Madison (1991), applying an ANCOVA does not always make groups equal. Furthermore, using a no-difference null hypothesis based on pretest scores is faulty logic as any fail-to-reject decision when testing any Ho does not justify believing that the null hypothesis is true (Huck & Cormier, 1996).
Other uncontrolled threats to validity include instrumentation, differential statistical regression, and the interaction of selection and history (Cook & Campbell, 1979). These threats have been described earlier and thus warrant only mention here.
The second class of quasi-experimental designs is the cohort designs. Cohort designs are typically stronger than nonequivalent-groups design because cohorts are more likely to be closer to equal at the outset of the experiment (Heppner et al., 1992). An example of a cohort in this context would be, TAMU freshman in 1995 versus TAMU freshman in 1996. For an example from the counseling literature, see Hogg and Deffenbacher (1988). The basic cohort design is diagrammed as follows:
In this design, the O1 represents a posttest administered to one cohort, while O2 represents a posttest administered to the following cohort. Even though the posttests occur at different points in time, the posttests do occur at the same point in the progression of the cohort (Cook & Campbell, 1979).
The most obvious problems with this design deal with the passage of time between the two cohorts, and the nonrandom assignment of participants to the cohort. The differences within the cohort before the treatment can be confounding. The specific threats to internal validity include history, when a researcher has no control over what events might occur in one cohort versus the other; changes surrounding instrumentation, testing, selection, and many interactions with selection.
The reason why cohort designs are useful include the "quasi-comparability" (Cook & Campbell, 1979, p. 127) that can often be assumed between cohorts who do receive a treatment, and those who do not. Many times these cohort groups are more similar to each other than are experimental groups, especially with regard to certain demographics (Cook & Campbell, 1979).
Time series design
The third class of quasi-experimental designs is the time-series design. These designs are characterized by multiple observations over time (e.g., Kivligham & Jauquet, 1990) and involve the same participant observations to record differences attributed to some treatment, or similar but different participants. In the interrupted time-series design (the most basic of this class) a treatment is introduced at some point in the series of observations (Heppner et al., 1992), and this design is diagrammed as follows:
O1 O2 O3 O4 X O5 O6 O7 O8
The impetus for performing this design is to observe over time, any difference after the treatment is implemented to discern if there is a continuous effect versus a discontinuous effect. A continuous effect would be treatment effects that remain stable after the initial discontinuity produced by the intervention. A discontinuous effect would be a result that decays over time. This design also accounts for effects that are instantaneous, versus delayed in their manifestations (Cook & Campbell, 1979), i.e., with repeated observations after the treatment is implemented, the researcher can ascertain how quickly the effects are initiated.
The major threat to internal validity in this design is history, i.e., that variables other than the treatment under investigation came into play immediately after introduction of the treatment, e.g., a seasonal effect (the weather in a study on absenteeism from work) (Heppner et al., 1992). Also, sometimes instrumentation changes with the passage of time, as does selection (or attrition). Selection is a plausible threat when there is a differential attrition rate, after introduction of the treatment (Cook & Campbell, 1979).
Summary and Conclusions
In counseling research, which is often applied, the utility of the findings are greatly reduced if the findings may be attributable to something other than the treatment under observation. We have examined the major threats to validity and their resultant effects in the context of using different research designs. Every experiment is imperfect from the standpoint of the final interpretation, and the attempt to "fit" the results into a developing science (Campbell & Stanley, 1963). The previous discussion of validity and designs was meant to guide researchers to use better designs when developing their studies, to increase awareness of the residual imperfections in their particular design to help account for alternative interpretations of the results. Hopefully, the call for "better and improved" science has been supported through this paper, and this reading will entice other students and researchers to contemplate long and hard before settling on a research design.
Bateman, T. S., Sakano, T., & Fujita, M. (1992). Roger, me, and my attitude: Film propaganda and cynicism toward corporate leadership. Journal of Applied Psychology, 77, (5), 768-771.
Benjamin, L. (Ed.). (1988). A history of psychology. New York: McGraw-Hill.
Birk, J. M., & Brooks, L. (1986). Required skills and training needs of recent counseling psychology graduates. Journal of Counseling Psychology, 33, (3), 320-325.
Braaten, L. J. (1989). The effects of person-centered group therapy. Person Centered Review, 4, (2), 183-209.
Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis for field settings. Chicago: Rand McNally.
Ford, D. H. (1984). Reexamining guiding assumptions: Theoretical and methodological assumptions. Journal of Counseling Psychology, 31, (5) 461-466.
Gannon, T., & Deely, J. (Eds.). (1991). Shaping psychology: How we got where we're going. In J. Deely & B. Williams (Series Eds.), Sources in semiotics, (Vol. 10). Lanham, MD: University Press of America.
Gelso, C. J. (1979b). Research in counseling: Clarifications, elaborations, defenses, and admissions. The Counseling Psychologist, 8, (3), 61-67.
Gelso, C. J. (1979a). Research in counseling: Methodological and professional issues. The Counseling Psychologist, 8, (3), 7-36.
Goldfried, M. R. (1984). Training the clinician as scientist-professional. Professional Psychology: Research and Practice, 15, 477-481.
Goldman, L. A. (1976). A revolution in counseling research. Journal of Counseling Psychology, 23, (6), 543-552.
Hains, A. A., & Szyjakowski, M. (1990). A cognitive stress-reduction intervention program for adolescents. Journal of Counseling Psychology, 37, (1), p. 80.
Heppner, P. P, Kivlighan, D. M., & Wampold, B. E. (1992). Major research designs. In C. Verduin (Ed.), Research design in counseling (pp. 115-165). Pacific Grove, CA: Brooks-Cole.
Hiebert, B., Simpson, L., & Uhlemann, M. R. (1992). Professional identity and counselor education. Canadian Journal of Counseling, 26, 201-208.
Hill, C. E., & Corbett, M. M. (1993). A perspective on the history of process and outcome research in counseling psychology. Journal of Counseling Psychology, 40, (1), 3-24.
Hogg, J. A., & Deffenbacher, J. L. (1988). A comparison of cognitive and interpersonal-process group therapies in the treatment of depression among college students. Journal of Counseling Psychology, 35, (3), 304-310.
Howard, G. S. (1982). Improving methodology via research on research methods. Journal of Counseling Psychology, 29, (3), 318-326.
Huck, S. W. (1991). True experimental design. Journal of Experimental Education, 59, (2), 193-196.
Huck, S. W., & Cormier, W. H. (1996). Principles of research design. In C. Jennison (Ed.), Reading statistics and research (2nd ed., pp. 578-622). New York: Harper Collins.
Jemmott, L. S., & Jemmott, J. B. (1992). Increasing condom-use intentions among sexually active black adolescent women. Nursing Research, 41, (5), p. 273.
Kelley, J., Smits, S. J., Leventhal, R., & Rhodes, R. (1970). Critique of the designs of process and outcome research. Journal of Counseling Psychology, 17, (4), 337-341.
Kivlighan, D. M., & Jauquet, C. A. (1990). Quality of group member agendas and group session climate. Small Group Research, 21, (3), 205-219.
Kush, K., & Cochran, L. (1993). Enhancing a sense of agency through career planning. Journal of Counseling Psychology, 40, (4), 434-439.
Laser, E. D. (1984). The relationship between obesity, early recollections, and adult life-style. Individual Psychology: Journal of Adlerian Theory, Research & Practice, 40, (1), 29-35.
Lofton, L. B., & Madison, S. Q. (1991). The extreme dangers of covariance corrections. In B. Thompson (Ed.), Advances in educational research: Substantive findings, methodological developments (Vol. 1, pp. 133-147). Greenwich, CT: JAI Press.
Lysynchuk, L. M., Pressley, M., dAilly, H., Smith, M., & Cake, H. (1989). A methodological analysis of experimental studies of comprehension strategy instruction. Reading Research Quarterly, 24, (4), 458-472.
Magoon, T. M., & Holland, J. L. (1984). Research training and supervision. In R. W. Lent & S. D. Brown (Eds.), Handbook of counseling psychology (pp. 682-715). New York: Wiley.
OHear, M. F., & MacDonald, R. B. (1995). A critical review of research in developmental education: Part I. Journal of Developmental Education, 19, (2), 2-4, 6.
Parker, R. M. (1993). Threats to the validity of research. Rehabilitation Counseling Bulletin, 36, (3), 131-138.
Schulz, D. P. (1972). A history of modern psychology. San Diego: Academic Press.
Serlin, R. C. (1987). Hypothesis testing, theory building, and the philosophy of science. Journal of Counseling Psychology, 34, (4), 365-371.
Serlin, R. C., & Lapsley, D. K. (1985). Rationality in psychological research: The good-enough principle. American Psychologist, 40, 73-83.
Thompson, B. (1994, April). Common methodology mistakes in dissertations, revisited. Paper presented at the annual meeting of the American Educational Research Association, New Orleans. (ERIC Document Reproductive Service No. ED 368 771)
Van Noord, R. W., & Kagan, N. (1976). Stimulated recall and affect simulation in counseling: Client growth reexamined. Journal of Counseling Psychology, 23, (1), 28-33.
Watkins, C. E., Lopez, F. G., Campbell, V. L., & Himmell, C. D. (1986). Contemporary counseling psychology: Results of a national survey. Journal of Counseling Psychology, 33, (3), 301-309.
Whitley, J. M. (1984). A historical perspective on the development of counseling psychology as a profession. In S. D. Brown & R. W. Lent (Eds.), The handbook of counseling psychology (pp. 3-55). New York: Wiley.
Wooley, K. K., Duffy, M., & Dawson, T. E. (1996, August). Developmental themes in the clinical judgments of counseling psychologists. Poster session presented at the annual meeting of the American Psychological Association, Toronto, Canada.
A poem by Schuyler W. Huck (1991):
True Experimental Design
R O1 X O2
R O1 O2
There once was a research design
That looked, on first glance, oh so fine
Yet it is stupidity
To view its validity
As chaste due to Rs on each line
For one thing, mere randomization
That's done to gain equalization
Of Es versus Cs
Gives no guarantees
That you'll have good gen'ralization
Results may, for instance, show all
That Es topped the Cs (with p small)
But if that full troop
Aint your "target" group
Wide claims will lead to your downfall
Or X may work well for just some
While others to it are quite numb
The treatment's true fate
Depends on the trait
One needs to gain from X shall come
The one who rates subjects may know
Which S's got treatment-and so
May ruin it all
And truth from your study won't flow
If treatment is novel, each E
May first say "It's super for me!"
But as time goes by
E may think and sigh
"No value in X do I see"
Or just the reverse can occur
With X thought at first to deter
But once acclimated
E's may be elated
And to X high rank they'll confer
Should hist'ry and X interact
'Tis true there is no artifact
But if replicated
You'll be decimated
By findings that prove inexact
The IV or DV may be
Reported not sufficiently
If later one tries to
Your study to redo
New findings may be what you'll see
If two or more studies are done
With S's not used in the just one
A treatment...it's "grade"
May strengthen or fade
If only one study is run
A pretest or posttest may make
The E group to X wide awake
O leads to creation
And sans O the treatment won't take
The time from the X to O2
Is worthy of careful review
For if it's revised
When new plans devised
Fresh findings may seem quite askew
The subjects may be told or know
Observers are watching...and so
They'll work with great skill
The watchers stop watching and go
Or what if the subjects perceive
What you hope and want and believe
Unconscious or not
Like robots they trot...
And data you want you'll receive
We hate now to sound so paternal
But we must move past threats "external"
More pitfalls there are
To rip, maim, and scar
And keep your work out of a journal
Suppose X or the O thought
To deal with a construct that's "caught"
When such is not true
With "fit" that's askew
One's main claims won't be what they ought
Moreover, one should not assume
That stat work can't add to the gloom
Assumptions may be
False prima facie
And that makes the truth not illume
Or what if theres meager precision?
Strong Xs you'll miss for poor "vision"
The Error Type II
Will hide truth from view
And you'll make an inapt decision
But what of those "internal" threats?
Are all tamed, like nicely trained pets?
If that rings true
You've learning to do
For R's catch just some in their nets
For instance, take treatment diffusion
Which can make lots of confusion
Should E give to C
All X that there be
You'll come to a faulty conclusion
Or what if the folks in the C group
Feel they were assigned to the "B" group
And you victimized
By "findings" as murky as pea soup
The converse may also take place
C's saying, "We will win the race!"
Each tries like a kid
(John Henry once did!)
Yet this can one's findings debase
And what if there's subject attrition
Related to the X condition?
It may just appear
That X was "in gear"
When that's just a sad apparition
Of course there is further the chance
That someone may try to enhance
The plight of those in
The C group wherein
Such help is a foul circumstance
And lastly consider the stew
You're in if on post E's construe
On DV their standing
As not so commanding
Cause X brought respect for O2
By "lastly" we sure don't intend
To mean that one need not attend
To problems that have now
No label, yet somehow
They could wreck one's work in the end
The moral, one hopes, is quite clear
The word "true" does not make it appear
That problems of import
Are all on the backcourt
When really they may lurk quite near!!!
©1999-2012 Clearinghouse on Assessment and Evaluation. All rights reserved. Your privacy is guaranteed at