Texas A&M University, January 1997

The present paper reviews the basic processes of exploratory factor analysis (EFA) as regards evaluating test score validity. It is emphasized in this paper as well as elsewhere that scores (not tests) vary in degrees of validity (Thompson, 1994). Construct validity is the focus of the paper, as there has been an increasing evolution toward a more unified view of validity (Shepard, 1993). The similarity between the processes of construct validation and exploratory factor analysis are described and the utility of EFA as a tool to evaluate score validity are explored.

Psychology evolved from the disciples of philosophy and physiology. As Weiten (1992) noted, "Philosophy provided the attitude, and physiology contributed the [scientific] method" (p. 3). Inherent in the scientific method, which is based on observation and measurement, are numerous statistical procedures.

One of the discoveries of psychology that researchers continue to explore and analyze with these statistical procedures addresses the differences between the right and left brain hemispheres. A popularly held suggestion is that the hemispheres exhibit different modes of thinking. It is suggested that the left hemisphere predominantly thinks in verbal terms, is analytic, rational, logical and linearly oriented. The right hemisphere’s modes of thinking include nonverbal, nonrational, intuitive and holistic approaches (Weiten, 1992).

The connection between the statistical procedures utilized by psychologists in their research and right vs. left brain functioning, is that, while the content can be processed in either hemisphere, some professsors teach the material in a technical manner amenable to those persons informally known as "left-brainers". Other teachers present statistical concepts in an intuitive way. This approach appeals to the "right-brainers" in the world (or in this case, the statistics courses)! As readers of this article may have surmised, the author of this paper falls into the latter category. For this reason, the present statistically-oriented paper is written from a right hemisphere perspective, i.e., conceptually and intuitively.

This paper came about as a result of a literature review on the topic of substance abuse, this author's primary area of interest. In the process, it became clear that a new instrument for determining alcohol dependence may be warranted. The decision was made to pursue the undertaking of creating such an instrument. One of the specific questions to be answered as part of the undertaking was, "Will this instrument measure what it is intended to"? It was suggested that an understanding of the concepts of construct validity and factor analysis would aid in answering the stated question. Furthermore, it was hinted that conducting a factor analysis would specifically address construct validation, and in the process, prove a useful tool in evaluating score validity.

This paper, therefore, will address both construct validity and factor analysis. Specifically, both concepts will be defined and their processes highlighted. In the discussion concerning validity, it will be emphasized that it is the inferences from scores of tests that are (or are not) valid, not the tests themselves. Finally, it will be shown that exploratory factor analysis is a tool for use in the evaluation of score validity, particularly in reference to construct validity.

A statistics professor of this author has frequently noted that a great many issues in statistical analyses are designed to confuse graduate students. This holds true regarding the definitions of many concepts. Factor analysis is an example of a topic that has been defined in a variety of ways. Reyment and Joreskog (1993) stated:

Cureton and D'Agostino (1983) described factor analysis as "a collection of procedures for analyzing the relations among a set of random variables observed or counted or measured for each individual of a group" (p. 1). The purpose, they said, "is to account for the intercorrelations among n variables, by postulating a set of common factors, considerably fewer in number than the number, n, of these variables" (p. 2). Bryman and Cramer (1990) broadly defined factor analysis as "a number of related statistical techniques which help us to determine them [the characteristics which go together]" (p. 253).

Gorsuch (1983) reminded the reader that "all scientists are united in a common goal: they seek to summarize data so that the empirical relationships can be grasped by the human mind" (p. 2). The purpose of factor analysis, he said, "is to summarize the interrelationships among the variables in a concise but accurate manner as an aid in conceptualization" (p. 2).

These definitions most likely make a great deal of sense to those "left-brained" individuals who understand complex things fairly easily. Kerlinger (1979) gave both a left-brained and a right-brained definition of factor analysis. For the left-brainers: "Factor analysis is an analytic method for determining the number and nature of the variables that underlie larger numbers of variables or measures" (p. 180). And for the right-brainers he noted: "It [factor analysis] tells the researcher, in effect, what tests or measures belong together--which ones virtually measure the same thing, in other words, and how much they do so" (p. 180). He further commented on factor analysis in terms of curiosity and parsimony. He noted, "Scientists are curious. They want to know what's there and why. They want to know what is behind things. And they want to do this in as parsimonious a fashion as possible. They do not want an elaborate explanation when it is not needed." (p. 179). He sounds like a very right-brained individual!

Each definition of factor analysis has common elements. Each refers in some way to the correlations among variables as reflected by the use of the words interrelationships, intercorrelations and relations. Further, each definition makes clear the notion of reducing the number of variables into a smaller set of factors. In short, factor analysis helps to explain things by reducing large amounts of information into a manageable form and size. Now that is an explanation that right-brained individuals (and of course, lefties, too), can comprehend!

Having been offered a basic understanding of what factor analysis is, the next question addresses validity. Specifically acknowledged is construct validity and what the notion of validity has to do with factor analysis.

Cronbach (1971) discussed validation as a process used by a test developer or test user to collect evidence that supports the types of inferences to be from test scores. Different aspects of validity have been defined. Crocker and Algina (1986) discussed three types of validation studies conducted to gather evidence of the usefulness of scores in addressing a specified inference. Content validity studies are used to assess whether the items on an inventory or test adequately represent the construct of specific interest. In other words: Can the researcher draw an inference from an exmainee's test score to a larger domain of items like those that are on the test itself? Criterion-related validity, encompassing both predictive validity and concurrent validity, is studied in situations where a test user wants to draw an inference about a person's test score to performance on a real behavioral variable that has practical importance. Construct validity is studied when "the test user desires to draw an inference from the test score to performances that can be grouped under the label of a particular psychological construct" (Crocker & Algina, 1986, p. 218).

Four types of validity were identified by the American Psychological Association (APA) when validity standards were first codified in 1954. These four types corresponded to different aims of testing: (a) content validity (b) predictive validity, (c) concurrent validity, and (d) construct validity (American Psychological Association, 1954). In 1966 the APA reduced predictive validity and concurrent validity to a single category: criterion-related validity (American Psychological Association, 1966).

It has been suggested that construct validity encompasses both criterion and content validity. Sheperd (1993) noted that construct validity envelopes the empirical and the logical requirements of criterion and content validity. Anastasi (1986) agreed that construct validity subsumes both content validity and criterion-related validity requirements.

Nunnally (1978) reported that "construct validity has [even] been spoken of as ... 'factorial validity' " (p. 111). As much as 50 years ago, this concept was acknowledged by Guilford (1946): "The factorial validity of a test is given by its loadings in meaningful, common, reference factors. This is the kind of validity that is really meant when the question is asked: Does this test measure what it is supposed to measure?" (p. 428, emphasis added).

Again, 44 years later, Bryman and Cramer (1990) noted that "factor analysis enables us to assess the factorial validity of the questions which make up our scales by telling us the extent to which they seem to be measuring the same concepts or variables" (p. 253).

A noteworthy emphasis of the present paper focuses on construct validity, and so that concept will be further addressed. Construct validity, although defined in the previous section, was explained in a manner somewhat more amenable to the right-brained population by Heppner, Kivlighan and Wampold (1992). This definition of construct validity focused on how well the variables chosen by the researcher to represent a hypothetical construct really "capture the essence" (p. 46) of that hypothetical construct. Stated differently, construct validity is " the degree to which the measured variables used in the study represent the hypothesized constructs" (p. 47). Stated even more simply, construct validity answers the question: Does this test or instrument really measure what it is intended to measure?

It should be apparent at this point, to all persons, regardless of brain hemisphere dominance, that factor analysis has long been associated with construct validity. It follows then, that factor analysis is a useful tool with which to evaluate score validity.

Frequently, both lay persons and professionals can be heard commenting about tests being reliable or valid. Even the right-brained population can clearly see the error in such a statement. A test is just a test. Validity and reliability are functions of the test scores, as determined by the test takers. In addition, as pointed out by Sheperd (1992), "Validity must be established for each particular use of a test" (p. 406). Cronbach (1971) agreed: "One validates, not a test, but an interpretation of data arising from a specified procedure" (p. 447).

Crocker and Algina (1986) described a process used to provide the construct validity of an instrument. In addition, they described four procedures (one being factor analysis) frequently utilized in construct validation. Regardless of the specific technique used, the steps generally followed include (a) formulating a hypothesis about how those who differ on the proposed construct do in fact differ in relation to other constructs already validated, (b) selecting or developing a measurement instrument that consists of items specifically representing the construct, (c) gathering empirical data so the hypothesized relationships can be tested, and (d) determining if the data are consistent with the hypothesis.

Heppner, Kivlighan, and Wampold (1992) delineated steps to be taken in factor analysis that curiously resemble those stated above as they apply to construct validation: (a) the researcher must first carefully think about the specific research question he or she wishes to address, (b) he or she chooses to use or develop an instrument constituting the variables specified, (c) the researcher selects the sample, collects the data, and begins to factor analyze the data in order to identify the common dimensions of a set of variables and to see which items go together to make up a factor, and (d) the researcher determines if the factors are correlated. See? It's starting to come together. We're finding out: Are the test items measuring what they're supposed to be measuring? Construct validity and factor analysis constitute a natural pairing.

It becomes evident (even to those with the heaviest of right brains) that factor analysis applies to construct validity. It should be clear at this point that one of the purposes of factor analysis is to determine the factors that underlie a given set of variables. In addition, the reader has hopefully been able to establish the connection between factor analysis and its usefulness as a tool in evaluating score validity. In other words, conducting a factor analysis of the observed scores on a given instrument, one can determine if indeed, the test is measuring the variables it purports to. This, in essence, is the definition of construct validation.

As has been shown regarding many other topics in statistics, various definitions exist for both exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). A sound left-brained definition is provided by Stevens (1996) for each of these two types of factor analysis:

Fortunately, Stevens (1996) also provided a table, giving a visual representation of the above definition (a definite plus for the right-brainers):

In the process of determining whether the identified factors are correlated, EFA answers the question asked by construct validity: Do the scores on this test measure what the test is supposed to be measuring via addressing whether or not the factors are correlated? Attention is next turned to the issue of the specific process of exploratory factor analysis.

Before entering into a discussion as to the process of factor analysis, it is necessary to touch upon a fundamental assumption upon which these procedures are based. Without going into a lengthy explanation of the concept of linearity, suffice it to say that:

Factor analysis assumes that the observed (measured) variables are linear combinations of some underlying source variables (or factors). That is,

As noted previously, exploratory factor analysis can be used as a method of determining the minimum number of underlying hypothetical factors that represent a larger number of variables. In exploratory factor analysis, this is done by showing the intercorrelations among the variables without having prior specifications of what these factors might be.

Factors have not, up to this point, been specifically defined. There are, again, many definitions provided, enough to please both left- and right-brained individuals. Among them are Cureton and D'Agostino's (1983) definition: "The factors are random variables that cannot be observed or counted or measured directly, but which are presumed to exist in the population and hence in the experimental sample .... they are sometimes termed latent variables" (p. 3). Tinsley and Tinsley (1987) stated that factors are "hypothetical constructs or theories that help interpret the consistency in a data set" (p. 414). Kline (1994) defined a factor as a "dimension or construct which is a condensed statement of the relationship between a set of variables" (p. 5). Kim and Meuller's (1978) definition stated that factors are "hypothesized, unmeasured, and underlying variables which are presumed to be the sources of the observed variables ... which are smaller in number than the number of observed variables, [and] are responsible for the covariation among the observed variables" (pp. 77, 12). Cureton and D'Agostino (1983) clarified the hypothetical nature of the factors:

The factors are actually hypothetical or explanatory constructs. Their reality in the individuals of the population or sample is always open to argument. At the conclusion of a factor analysis we can only say of the factors that if they were real, then they would account for the correlations found in the sample. (p. 3)

In essence, then, factors are the latent (unobserved), hypothetical, underlying concepts (or constructs) deduced from the correlations between the measured variables of the instrument or test. (Notice the term construct used in the definition of factor; no wonder the association between exploratory factor analysis and construct validity).

Data matrix

The first step in an exploratory factor analysis is to display the data in a data matrix. A data matrix is "any array of numbers with one or more rows and one or more columns" (Reymont & Joreskog, 1993, p. 15). This appears to be quite straightforward (much to the surprise and relief of the right-brained). Ah, but not so fast. In an effort to complicate matters, there are issues of a vector (a matrix that has only one row) and a scalar (which has both one row and one column), as well as a variety of matrices identified by Gorsuch (1983) in developing factor analytic concepts. (The right-brained among you are possibly noticing a constriction of air passages at the number of possible options, but not to worry. This is merely an introductory paper on the topic of factor analysis.)

Correlation matrices.

In order to determine the factors underlying the variables, a "variable reduction scheme" (Gorsuch, 1983, p. 362) is used which shows how the variables cluster together; i.e., the variables are correlated with one another. These correlations are represented in a matrix of association. A statistical measure of association such as the Pearson r is used to indicate the magnitude of the correlations.

A correlation (or variance-covariance) matrix represents the relationships among the set of variables in the study. In this correlation (or variance-covariance) matrix of variables, the values located on the diagonal will be 1.0. This is because each of the variables will correlate perfectly with itself. The off-diagonal elements are the co-variances between all variable pairs. (Remember, right-brainers, this simply means the correlations between the variables.)

Because the number of correlations in the matrix reflects the number of variables used in a study, it is possible that a single correlation matrix may have thousands of entries. Factor analysis, explained Hetzel (1995), "attempts to simplify the correlation matrix by accounting for a large number of relationships with a smaller number of explanatory constructs [i.e., factors]" (p. 7). He further stated that these hypothetical factors are determined by examining additional data matrices, specifically the factor pattern matrix and the factor structure matrix.

In much of the literature on factor analysis, the term "factor loading" is used instead of the more accurate terms, factor pattern coefficients and factor structure coefficients, which are the elements comprising the factor pattern and factor structure matrices. The exact nature of these coefficients and corresponding matrices is beyond the scope of this paper. The important element is that factor pattern coefficients represent the relationship of a specific variable to a specific factor without the influence of other variables (Stevens, 1992). The factor structure coefficients can be thought of as being identical to structure coefficients in other types of correlational analyses. These coefficients show the correlations of the variables with the factors (Hetzel, 1995). It is with the results of these additional matrices, and through the careful interpretation of the data, that the factors are extracted and interpretations made.

We are reminded by Cattell (1978) that "factor analysis is, in principle, nothing more than asking what the common elements are when one knows the correlation" (p. 20). It is at this point, when we have calculated the correlations between the variables and factors, that we can begin to determine the number of factors underlying the variables. The chief concern, at this stage, according to Kim and Meuller (1978) is whether a smaller number of factors can account for the covariation among the original, larger set of variables. Gorsuch (1983) indicated that there are numerous methods that can be used in deciding how many factors to retain. Again, these methods are too detailed for the current paper, but in general, regardless of the method used, he suggested that "one would want to account for at least 70% of the total variance" (p. 367).

The critical point in deciding how many factors to retain is that this decision requires the researcher to carefully consider the data and to use his or her judgment. As with many other statistical concepts, a number of decision rules are available to help guide the researcher with the decision as to the specific number of factors to retain. This topic was summarized by Hetzel (1995):

Following the initial extraction of factors, an interpretation of these factors is necessary. Kim and Meuller (1978) pointed out:

It should be noted that the factors must be called something other than the name of a particular observed variable. The reason for this is that factors are latent aggregates of observed variables and the factor name should represent the aggregate and not be confused with a specific measured variable.

At this point in the analysis, the minimum number of factors that can account for the observed correlations have been identified and named. To obtain a more easily interpretable solution regarding the factors, the researcher can engage in a process known as rotation. This is most easily done by computer and again, is too complicated a matter for this paper. The results of rotation, however, indicate "the simplest solution among a potentially infinite number of solutions that are equally compatible with the observed correlations" (Kim & Mueller, 1978, p. 59).

The process of exploratory factor analysis results in the smallest, and most compatible number of underlying factors from a larger set of initial variables on a test or instrument. The process can be summarized as follows: (a) the researcher collects observed scores (raw data) on an instrument without having a preconceived notion as to the number of underlying factors, (b) presents this information in data matrices, (c) correlates the variables, and (d) identifies the factors underlying the variables.

Tests are not valid in and of themselves. Rather, test scores may be valid. Although many types of validity have been identified, construct validity has been suggested as encompassing all forms of validity. In addition, construct validity addresses the issue of whether a test does, in fact, measure what it purports to.

Exploratory factor analysis serves a number of functions including identification of the number of factors that underlie a set of variables and determination as to whether the factors are correlated or uncorrelated. This process can be an aid in evaluating the score validity of a test via these two functions.

American Psychological Association. (1954). Technical recommendations for psychological tests and diagnostic techniques. Psychological Bulletin, 51, 201-238.

American Psychological Association. (1966). Standards for educational and psychological tests and manuals. Washington, DC: Author.

Anastasi, A. (1986). Evolving concepts of test validation. Annual Review of Psychology, 37, 1-15.

Bryman, A., & Cramer, D. (1990). Quantitative data analysis for social scientists. New York: Routledge.

Cattell, R.B. (1978). The scientific use of factor analysis. New York: Plenum Press.

Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Fort Worth: Harcourt Brace Jovanovich College Publishers.

Cronbach, L.J. (1971). Test validation. In R.L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 443-507). Washington, DC: American Council on Education.

Cureton, E.E., & D'Agostino, R.B. (1983). Factor analysis: An applied approach. Hillsdale, NJ: Erlbaum.

Gorsuch, R.L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum.

Guilford, J.P. (1946). New standards for test evaluation. Educational and

Psychological Measurement, 6, 427-439.

Harman, H.H. (1960). Modern factor analysis. Chicago: The University of Chicago Press.

Heppner, P.P., Kivlighan, Jr., D.M. & Wampold, B.E. (1992). Research design in counseling. Pacific Grove, CA: Brooks/Cole.

Hetzel, R.D. (1995, April). The use of factor analysis in counseling psychology research: Current practice is not necessarily best practice. Paper presented at the annual meeting of the American Education Research Association, San Francisco, CA.

Kerlinger, F.N. (1979). Behavioral research: A conceptual approach. Dallas: Holt, Rinehart and Winston.

Kim, J.O., & Mueller, C.W. (1978). Introduction to factor analysis. Beverly Hills: Sage Publications.

Kline, P. (1994). An easy guide to factor analysis. London: Routledge.

Nunnally, J.C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.

Reymont, R., & Joreskog, K.G. (1993). Applied factor analysis in the natural sciences. New York: Cambridge University Press.

Shepard, L.A. (1993). Evaluating test validity. In L. Darling-Hammond (Ed.), Review of research in education (Vol. 19, pp. 405-450). Washington, DC: American Educational Research Association.

Stevens, J. (1992). Applied multivariate statistics for the social sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

Stevens, J. (1996). Applied multivariate statistics for the social sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.

Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54 (4), 837-847.

Tinsley, H.E.A., & Tinsley, D.J. (1987). Uses of factor analysis in counseling psychology research. Journal of Counseling Psychology, 34, 414-424.

Weiten, W. (1992). Psychology: Themes and variations (2nd ed.). Pacific Grove, CA: Brooks/Cole.