Lawrence M. Rudner
LMP Associates & the Maryland Assessment Research Center for Education Success

Unless a test is perfectly reliable and thus contains no error, every time we make a classification based on a test score, we should expect some number of misclassifications. We can expect that some examinees whose true ability is above the cut score will be incorrectly classified as non-masters (false negatives) and that some number of low-ability examinees will be incorrectly classified as masters (false positives).

This paper provides and illustrates a method to compute the expected number of misclassifications. This information can help policy makers decide whether the risks are sufficiently small or whether the costs for improvements are justified. One particularly useful application of this procedure is to estimate the probability of a true master consistently failing multiple test administrations. Another is to examine the impact of cut score adjustments. Web-based software to apply this method is available at  http://ericae.net/pare/misclass/class.asp. PC based software is available from the author.

Approach

I will develop the procedure using three-parameter item response theory and two state classifications (mastery and non-mastery). There are classical test theory analogs and the logic can easily be extended to more categories. I start with a test that maps individual scores (_i's) onto a continuous scale (a scale) and a cut score on that scale (_c) used to classify examinees into one of two discrete categories. Examinees whose scores are above the cut score will be classified as masters; those below as non-masters.

We make a distinction between the categories examinees should be placed in based on their true scores and the categories they are placed in based on observed score. The goal of this paper is to create, and then analyze, a two by two classification table, such as Table 1, indicating the expected proportions of correct and incorrect classifications.

In table 1, the upper left and lower right quadrants represent correct classifications; the other two quadrants represent incorrect classifications.

The expected proportion of all examinee classified as a masters that are true non-masters is:

(1)

It should be noted that the phrases true masters and true non-masters are statistical terms meaning that the true ability is above or below the arbitrarily set cut score. In (1), P( > _c | _i) is the probability of having an observed score, , above the cut score given a true score equal to _i, f (_i) is the expected number of people whose true score is _i and n is the total number of examinees. Thus P( > _c | _i ) f (_i) is the expected number of people whose true score is _i that will be classified as masters, i.e., will have an observed score greater than the cut score. We sum this value over all examinees whose true score is less than the cut score and divide by n to obtain the probability of being misclassified as a master (false positive).

Similarly, the expected proportion of false negative is:

Referring to (1), the probability of having an observed score above the cut score given _i, P( > _c | _i), is the area under the normal curve and to the right of

In (3), se(_i), the standard error of measurement evaluated at a score of _i, is

where I (_i) is the test information function evaluated at a score of _i. Lord(1980, pages 72-74) provides equations for I (_i) using weighted composite scoring, number right scoring, and item response theory scoring. Using IRT, the test information function at _i is the sum of the item information functions at _i which can be evaluated from the IRT a, b, and c item parameters.

The expected proportions of examinees whose true score is _i, f (_i)/n, can be estimated from the pilot sample. Denoting the probability of obtaining a score of _i as P(_i), and looking at as a continuous rather than a discrete variable, (1) and (2) become:

To complete the set of equations needed for our two-by-two classification table, the probability of correctly being classified as a master and the probability of correctly being classified as a non master are

If one assumes a normal distribution of mean µ and standard deviation , then P(_i) is the height of the Gaussian curve evaluated at (_i - )/.

Illustration

To illustrate an application of the above formulas, I generated a data set consisting of the item parameters for 50 items. The a, b, and c parameters were each normally distributed with means of 1.43, 0.00, and 0.20 respectively. Assuming a normally distributed examinee population with mean=0 and sd=1.0, the weighted average standard error is .393 which corresponds to a classical reliability coefficient of .92.

Applying equations (5) through (8) with a passing score of _c= -.25, which is about the 40th percentile, yields