Counseling psychology is a burgeoning field still in its inchoate stages, attempting to gain/maintain autonomy and respect. As students of a scientific-practicing counseling psychology program, it behooves us to conduct well thought-out, meaningful research in the name of practicing "good science," as does it benefit all counseling psychologists in the name of furthering the field's namesake. Unfortunately, many times the tendency to embark on a research endeavor lacks the necessary foresight in constructing the design. Research designs are pervious to many different types of threats to their internal and external validity. In the traditions of Campbell and Stanley, and Cook and Campbell, this paper will elucidate some of the more common types of research designs, along with the coexistent threats to validity. Further, an example of each type of design has been given from the counseling literature for the interested reader to paruse and help make the concepts concrete.

Poem by Skyler Huck, University of Tennessee
Upon its inception as an experimental science, psychology has utilized the scientific method found in the physical sciences. In 1879 Wilhelm Wundt opened the first psychological laboratory in Leipzig, Germany, and with that commencement also came the first arguments about the validity of Wundt's experiments (Benjamin, 1988). Since that time, the scientific method has been applied to various psychological constructs, e.g., behaviorism, learning theory, Gestalt psychology, animal experimentation, cognition, and functionalism (Gannon, 1991).

Counseling Psychology has experienced many "growing pains" in its attempts at being recognized as a separate (Hiebert, Simpson, & Uhlemann (1992); see also Wooley, Duffy, & Dawson, (1996) for a preliminary study to support counseling psychology's attempts at autonomy), viable science, which has been linked previously by some to the utilization of the scientific method (Hill & Corbett, 1993; Schultz, 1972). Since the inauguration of counseling psychology in 1946 (Whitley, 1984) it was mainly an applied psychology. At this juncture in time, the application of psychology was only beginning to gain respect from the intact group of psychologists who considered themselves as "pure"--that is they engaged in experimental psychology (Benjamin, 1988). In light of this zeitgeist and the identity struggles within the division, it stands to reason that counseling psychology places scientific inquiry through the rigor of the scientific method, as a core function.

In the past 20 years, there has been growing dissension in the ranks of counseling psychology researchers regarding the way in which research focusing on the philosophy of science and counseling psychology is being conducted. Many believe that researchers are placing too much emphasis on objectives and questions to which research should be directed, with little attention to actual research designs and methods (Ford, 1984; Goldman, 1976; Howard, 1982; Parker, 1993; Serlin, 1987; Serlin & Lapsley, 1985). Others have directly stated that more attention should be placed on the training aspects of research methodology (Gelso, 1979a, 1979b; Goldfried, 1984; Magoon & Holland, 1984), (note though that Birk and Brooks, 1986, report that 81% of 300 counseling psychologists surveyed reported adequate training in research). Indeed, the results of a national survey of counseling psychologists indicate that 49.6% of their time is devoted to the activity of research (Watkins, Lopez, Campbell, & Himmell, 1986), thus further supporting the relevance of the present paper to counseling psychologists.

If this is the case, how is it that so many counseling psychologists and counseling psychology students are producing "bad science"? For example, in a specific instance of reviewing one (psychology-type) department's recent dissertations, Thompson (1994) found numerous blatant mistakes due to methodological errors, and many others have challenged the current state of research (e.g., O'Hear & MacDonald, 1995). These errors are most likely representative and indicative of the more common mistakes found in research presently.

With this backdrop and apparent need for remediation, the present paper presents a "primer on experimental designs," with the specific goal of (review), and the more comprehensive intention that through "better and improved science", counseling psychology will continue to solidify its place as an indisputably separate and viable field. In the traditions of Campbell and Stanley (1963) and Cook and Campbell (1979), a review of experimental and quasi-experimental designs and how threats to validity impacts counseling research will be presented, employing examples from the current counseling literature.

Internal validity is one important type of research validity. The term "internal validity" refers to the extent that extraneous variables (error variance) in an experiment are accounted for. It is paramount to the researcher that model specification error variance (as distinct from measurement and sampling error variance) is controlled because if not, the researcher can not emphatically conclude that the observed outcome is due to the independent variable(s) (Parker, 1993). Campbell and Stanley (1963) stated that "internal validity is the basic minimum without which any experiment is uninterpretable" (p. 5). There are eight major threats to internal validity: (a) history, encompassing the environmental events occurring between the first and second observations in addition to the independent variable(s); (b) maturation, which refers to the processes within the participants (psychological and/or biological) taking place as a function of the passage of time, not attributable to the independent variable(s); (c) testing, which is sensitization to the posttest as a result of having completed the pretest; (d) instrumentation, which refers to deterioration or changes in the accuracy of instruments, devices or observers used to measure the dependent (outcome) variable; (e) statistical regression, which operates when groups are selected on the basis of their extreme scores, because these anomalous scores tend to regress toward the mean on repeated testing; (f) selection, which refers to the factors involved in placing certain participants in certain groups (e.g., treatment versus control), based on preferences; (g) mortality, which refers to the loss of participants and their data due to various reasons, e.g., death or sickness; and (h) interactions of previous threats with selection. For example, a selection-maturation interaction results when the experimental groups are maturing at different rates based on the selection of the participants (Campbell & Stanley, 1963). In later writings, Cook and Campbell (1979) identify an additional threat to internal validity. This is ambiguity about the direction of casual influence when all other plausible third-variable explanations have been ruled out of the A-B relationship, but it remains unclear as to whether A causes B, or B causes A.

This construct asks the question of generalizability. Which populations, settings, treatment variables and measurement variables can these results be generalized to? Generalizing across persons requires research samples to be representative of the population of interest. Generalizing across times and settings usually necessitates systematically administering the experimental procedure at different times and different settings (Parker, 1993). The inability to obtain samples that are representative of the populations from which they came, especially if studied in various settings, and at different times, results in the inability to generalize beyond the persons, time, and setting of the original study. Tests that do meet the representativeness criteria are, in essence, tests of statistical interaction. For example, if there is an interaction between a therapeutic treatment and ethnicity, then it can not be decisively stated that the treatment holds true across different ethnicities. When effects of differing magnitude exist, the researcher must delineate when and where the effect holds, and when and where it does not (Cook & Campbell, 1979).

The statistical interaction threats to external validity outlined by Cook and Campbell (1979) are as follows: Interaction of selection and treatment (as in the previous example dealing with ethnicity); and interaction of setting and treatment (e.g., can a casual relationship obtained on a military installation also be obtained on a university campus?). The last interaction is between history and treatment. In this case, the question involves to which period of the past or future can the results obtained be generalized. For example, the majority of experiments take place on university campuses, with undergraduate university students as participants. If an experiment was conducted on the day after a football loss to this university's arch rival, then the results may not generalize even to a week after the loss, much less beyond the participants and setting represented in the original study.

Parker (1993) reviewed and synthesized the Campbell and Stanley (1963) and the Cook and Campbell (1979) work and explicated two additional threats to external validity: The interaction of treatments with treatments, which refers to the administration of multiple treatments administered to the same participants, e.g., time-series designs wherein the effects may be cumulative; and the interaction of testing with treatment, not to be confused with the internal validity threat of testing when the pretest sensitizes the participant to the posttest. In the external validity case, the pretest may increase or decrease the participants responsiveness or sensitivity to the treatment.

The above description of the most common threats to internal and external validity lays the groundwork for the planning of research projects. With these potential pitfalls in mind, the researcher is now ready to begin to plan which treatment design will be implemented (Lysynchuk, Pressley, d'Ailly, Smith, & Cake, 1989). The following explanation of the different types of treatment designs and the inherent threats to their validity will use an "X" to represent the exposure of a group to an experimental treatment or event. An "O" will signify some type of observation or measurement. The Xs and Os in the same row will refer to the same group, and the order of the characters from left to right will designate the temporal order of the events. "R" will exemplify random assignment, if necessary (Campbell & Stanley, 1963).

In a review of the designs of the process and outcome studies published in the Journal of Counseling Psychology (JCP) between the years of 1964 through 1968, Kelley, Smits, Leventhal, and Rhodes (1970) found that 54% of the studies utilized a preexperimental design. Preexperimental designs are those in which there is no control group and/or have comparison groups that are formed nonrandomly, therefore yielding results which are difficult to interpret (Huck & Cormier, 1996). The three preexperimental designs presented by Campbell and Stanley (1963) are the one-shot case study, the one-group pretest-posttest design, and the static group comparison. We will examine these designs in the order given.

Much past research applied a design in which a single group was studied only once after a treatment was applied. These studies are diagrammed as follows:

X O

According to Kelley Smits, Leventhal, and Rhodes (1970) the preponderance of designs they reviewed in the JCP were one-shot case studies (31%). Campbell and Stanley (1963) refer to these studies as having "...such a total absence of control as to be of almost no scientific value" (p. 5). They go on to state that "securing scientific evidence involves making at least one comparison" (p. 6) ...and that "It seems well-nigh unethical at least at the present time to allow, as theses or dissertations in education, case studies of this nature" (p. 7). As these studies are practically unused today, we will examine the threats inherent in the one-shot case design below, when they are associated with other more commonly used designs.

This design is judged to be better than design one (Campbell & Stanley, 1976) and is a catalyst for understanding how many of the extraneous variables that threaten internal validity play out. The one-group pretest-posttest design can be reviewed by referencing Jemmett and Jemmett (1992), and is diagrammed as follows:

O₁ X O₂

In this design, history is one of the uncontrolled rival hypotheses, as the changes between O₁ and O₂ may have been due to events that possibly occured in addition to the experimenter's X. The longer the time that elapses between the two observations, and the more participants for which specific events happen collectively, then the more plausible history becomes as a rival hypothesis (Campbell & Stanley, 1963).

Other rival hypotheses include the participants maturing (physically or psychologically) between the pre and posttests, or possibly the participants do better on the posttest as a result of taking the pretest (testing). Maybe the measuring instrument changed over the course of the study (instrumentation), or certain participants may have selectively dropped out of the study (mortality/selection). If the participants scored atypically on the pretest, they may have regressed toward the mean naturally (statistical regression), without any influence of X (Huck & Cormier, 1996).

The third preexperimental design is the static-group comparison (e.g., Laser, 1984). In this design, a posttest is administered to two groups, one having been administered the X, and the other not (a control group). When diagrammed, this design appears as follows:

X O

O

The basic problem with this design is the unknown status of the two groups prior to the administration of X, since the participants are not randomly assigned to the two groups. If a difference is obtained at posttest, these results may have been the influence of X. Alternatively though, the difference could have been an initial difference between the two groups. Since the participants either self-select themselves for participation into either group, or two existing groups are used, this is a selection threat to internal validity. Another threat to internal validity in this design that might threaten, even if the groups began as equal, is the selective drop-out rate of the participants in one group (mortality) (Campbell & Stanley, 1963; Huck & Cormier, 1996).

True experimental designs yield results that are more trustworthy than the preexperimental designs due to the fact that random assignment is utilized, therefore reducing the amount of potential threats to internal validity (Huck & Cormier, 1996).

For an example of this design in practice, see Hains and Szyjakowski (1990) or Kush and Cochran (1993). The design is diagrammed as follows:

R O₁ X O₂

R O₃ O₄

Random assignment is employed to both groups, and both are given a pretest. One group is administered the X, and the other is not. A comparison of O₂ and O₄ should elucidate any effect of the X. The unique strength of this design is the addition of the pretest, though there are controversies surrounding the use of pretests after random assignment (Heppner, Kivligham, & Wampold, 1992). In this design, many of the previous internal threats to validity discussed so far are accounted for. The differences attributed to history (and hopefully instrumentation) between O₁ and O₂ would be similar to the differences between O₃ and O₄. The threat of maturation and testing should be equally manifested between the experimental and the control group, statistical regression, mortality, and selection interaction threats are protected by the random assignment of participants (Campbell & Stanley, 1963), occurring probably equally across the two groups.

Ironically, the major weakness of this design is in fact, its major strength, but for external validity reasons. The pretest would sensitize both the control group and the experimental group to the posttest in a like manner, therefore presenting no internal threat to validity. However, generalizing the results of a treatment that included a pretest in the design, to a different sample without a pretest, may yield much different results (Heppner et al., 1992).

Here, randomization is utilized to ensure the equalization of the two groups, without a pretest, as in the Van Noord and Kagan (1976) study. The design is depicted in this way:

Again, random assignment is employed to both groups. The X is administered to the experimental group, and the second group acts as a control. The internal validity of this design is basically solid. According to Cook and Campbell (1979), the posttest-only control group design is the prototypical experimental design, and most closely exemplifies a condition in which a casual relationship can be discerned between an independent and dependent variable.

The main weakness of this design concerns external validity, i.e., the interaction of selection and treatment (Campbell & Stanley, 1963). Because of random assignment, the selection of subjects is not supposed to present a threat to internal validity. Nonetheless, it is often unknown whether the results of the study would generalize to another population (Heppner et al., 1992). For example, there could potentially be great differences between the results of a course on speed reading taught to a graduate class, versus a speed reading course taught to a high school class. Another problem deals with the absence of a pretest employed to reduce variability in the dependent variable. Random assignment is thought by some to account for this preexisting variability. But according to Huck and Cormier (1996) random assignment is not always random because (a) many researchers have a very loose definition of what randomization is, and (b) true randomization carries with it very stringent criteria and many researchers are unaware of the necessary precision and falsely believe they have true randomization, when they do not. The suggestion given by Huck and Cormier is that researchers explain definitively just how they accomplished randomization.

When a pretest is desired, but there is concern over the effects of using a pretest, as in the Bateman, Sakano, and Fujita (1992) study, this design is used, notationally described as:

This design is a combination of the pretest-posttest control group design (the first two groups), and the posttest-only control group (the last two groups). The main purpose of this design is to account for potential effects of the pretest on the posttest, and lends some degree of future replicability. For example, the Solomon four-group design accounts for the problem that the pretest-posttest control group design has, by comparing O₂ to O₅ to account for pretest sensitization, the only difference being that O₂ receives a pretest prior to treatment. With regard to generalizability, the researcher can compare O₂ to O₄ and O₅ to O₆. If treatment effects are found in both cases, the results will be considered strong, and suggest future replicability, as one replication is confirmed with the data in hand (Heppner et al., 1992). The major drawback of this design is the amount of time, energy, and resources necessary to complete the study.

When a true experimental design is not available to a researcher for various reasons, e.g., in clinical settings where intact groups are already formed, when treatment can not be withheld from a group, or when no appropriate control or comparison groups are available, the researcher can use a quasi-experimental design. As in the case of the true experimental design, quasi-experiments involve the manipulation of one or more independent variables and the measurement of a dependent variable. The major difference between true and quasi-experimental designs is the random assignment of participants (Heppner et al., 1992). Therefore, the internal validity of the quasi-experimental design is higher than that of the pre-experimental design, but lower than the true experimental design (Huck & Cormier, 1996). There are three major categories of quasi-experimental design: the nonequivalent-groups designs, cohort designs, and time-series designs (Cook & Campbell, 1979). An example of each category will be given, though the reader should be aware that there are many variations of each of the following examples.

The nonequivalent-groups design is the most frequently used quasi-experimental design (Heppner et al., 1992; Huck & Cormier, 1996). This design is similar to the pretest-posttest control group experimental design considered earlier. The difference is the nonrandom assignment of subjects to their respective groups in the quasi-experimental design. The design is diagrammed as follows, and can be further perused by referencing Braaten (1989):

This design is one of most widely used designs in the social sciences because it is often interpretable. Cook and Campbell (1979) recommend this design when nothing better is available. The nonequivalent-groups design accounts for many of the threats to internal validity, except for four. The first uncontrolled threat is that of selection-maturation. As stated earlier, many researchers falsely believe that the administration of a pretest remedies the nonrandom assignment of participants, and use ANCOVA to "level" the groups. As has been succinctly pointed out by Loftin and Madison (1991), applying an ANCOVA does not always make groups equal. Furthermore, using a no-difference null hypothesis based on pretest scores is faulty logic as any fail-to-reject decision when testing any H_o does not justify believing that the null hypothesis is true (Huck & Cormier, 1996).

Other uncontrolled threats to validity include instrumentation, differential statistical regression, and the interaction of selection and history (Cook & Campbell, 1979). These threats have been described earlier and thus warrant only mention here.

The second class of quasi-experimental designs is the cohort designs. Cohort designs are typically stronger than nonequivalent-groups design because cohorts are more likely to be closer to equal at the outset of the experiment (Heppner et al., 1992). An example of a cohort in this context would be, TAMU freshman in 1995 versus TAMU freshman in 1996. For an example from the counseling literature, see Hogg and Deffenbacher (1988). The basic cohort design is diagrammed as follows:

In this design, the O₁ represents a posttest administered to one cohort, while O₂ represents a posttest administered to the following cohort. Even though the posttests occur at different points in time, the posttests do occur at the same point in the progression of the cohort (Cook & Campbell, 1979).

The most obvious problems with this design deal with the passage of time between the two cohorts, and the nonrandom assignment of participants to the cohort. The differences within the cohort before the treatment can be confounding. The specific threats to internal validity include history, when a researcher has no control over what events might occur in one cohort versus the other; changes surrounding instrumentation, testing, selection, and many interactions with selection.

The reason why cohort designs are useful include the "quasi-comparability" (Cook & Campbell, 1979, p. 127) that can often be assumed between cohorts who do receive a treatment, and those who do not. Many times these cohort groups are more similar to each other than are experimental groups, especially with regard to certain demographics (Cook & Campbell, 1979).

The third class of quasi-experimental designs is the time-series design. These designs are characterized by multiple observations over time (e.g., Kivligham & Jauquet, 1990) and involve the same participant observations to record differences attributed to some treatment, or similar but different participants. In the interrupted time-series design (the most basic of this class) a treatment is introduced at some point in the series of observations (Heppner et al., 1992), and this design is diagrammed as follows:

O₁ O₂ O₃ O₄ X O₅ O₆ O₇ O₈

The impetus for performing this design is to observe over time, any difference after the treatment is implemented to discern if there is a continuous effect versus a discontinuous effect. A continuous effect would be treatment effects that remain stable after the initial discontinuity produced by the intervention. A discontinuous effect would be a result that decays over time. This design also accounts for effects that are instantaneous, versus delayed in their manifestations (Cook & Campbell, 1979), i.e., with repeated observations after the treatment is implemented, the researcher can ascertain how quickly the effects are initiated.

The major threat to internal validity in this design is history, i.e., that variables other than the treatment under investigation came into play immediately after introduction of the treatment, e.g., a seasonal effect (the weather in a study on absenteeism from work) (Heppner et al., 1992). Also, sometimes instrumentation changes with the passage of time, as does selection (or attrition). Selection is a plausible threat when there is a differential attrition rate, after introduction of the treatment (Cook & Campbell, 1979).

In counseling research, which is often applied, the utility of the findings are greatly reduced if the findings may be attributable to something other than the treatment under observation. We have examined the major threats to validity and their resultant effects in the context of using different research designs. Every experiment is imperfect from the standpoint of the final interpretation, and the attempt to "fit" the results into a developing science (Campbell & Stanley, 1963). The previous discussion of validity and designs was meant to guide researchers to use better designs when developing their studies, to increase awareness of the residual imperfections in their particular design to help account for alternative interpretations of the results. Hopefully, the call for "better and improved" science has been supported through this paper, and this reading will entice other students and researchers to contemplate long and hard before settling on a research design.

A poem by Schuyler W. Huck (1991):