VPI Occasional Paper: The Trouble With Headaches-- A Horrible Romance by Robert W. Bowman, Jr. and Robert B. Frary Virginia Polytechnic Institute and State University [Robert Bowman is no longer affiliated with VPI.] Linda and Kevin met in the hall after Dr. Johnson's class. Kevin could tell that Linda was worried--probably about the multiple-choice test They had just finished. Linda was comparing her answers with the answer key given out after class. Sympathetically, Kevin asked, "What's the problem, Linda?" "I studied for this test and I never get less than Bs, but I don't think I got over 60%," Linda said with dismay. "I did have a headache, but that couldn't have made me do this badly!" "It'll be okay; Dr. Johnson said he'll give the same numbers of each letter grade regardless of the scores," soothed Kevin. "That's the way all the profs did it at Michigan." "Well, this is Virginia Tech, so maybe you'd better explain some more." --------------------------------------------------- What's happening here? Think back over the tests you've taken in college. Do they remind you of a dog's breakfast--lots of soft, easy questions, some tougher, chewy questions, and a few indigestible lumps? But mostly soft. . . ? After all, we grew up taking spelling tests where a score of 70% meant we knew about 70% of the words on a long list from which the test words were a representative sample. Unfortunately, this simple approach to testing breaks down even in elementary school. How could you get a list of all the questions that could be asked in a history course? And even if you could, selecting randomly from the list probably wouldn't make a satisfactory test. In fact, just as in making a dog's breakfast, most college teachers don't really choose how to construct tests. They just sit down and write the best questions they can to cover what they have taught. Eventually, with experience, they learn to make their tests about as difficult as they intend. For many, that means that a student who scores 70% will receive a D. Since 70% is absolutely arbitrary, we should question whether it is desirable. In the discussion to follow, it is argued that a 70% passing level is _much_ _too_ _high_ for a large majority of college tests. There are exceptions of course; this claim does not apply to essay tests or tests designed to measure a specific skill. What it does concern are tests covering fairly diverse topics on which scores are determined by adding up points for each right answer. Almost any multiple-choice midterm or final examination would be in this category as would most short-answer and problem solving tests. Since these tests may be as hard or as easy as the teacher happens to make them, it is clear that the percentage of right answers does not estimate some level of achievement directly as do elementary spelling tests. What the scores do provide is a ranking of students in terms of their achievement in the area represented by the questions. If nearly all the scores are in the range of 35 to 50 points (or 70% to 100%), there are only 15 points over which to rank the students for the purpose of determining at least four letter grades, D through A. At each cut point, there will be a number of students who narrowly gain the higher grade or barely miss it. A greater spread of scores may be obtained if the test is made much harder. Those who made 35 to 50 on the easy test may have score ranging from 20 to the high forties on a much harder test. Then there will be many fewer students with scores adjacent to each cut point for determining letter grades. This spreading of scores minimizes the effect of minor scoring errors or variation in grading leniency. Assuming that the same proportion of A's, B's. C's, _etc_., are given as before the result is a much more accurate assessment of achievement. Think of all of this in terms of Linda's headache. Suppose she missed two questions she would not have had she felt better. With the score spread out, it is less likely that the two-point loss carried her over a grade boundary. Suppose Dr. Johnson had a headache too and made a lot of two-point errors in scoring the papers! Again, his mistakes are much less likely to affect grades if the cut points are far apart. At this point, the clever reader will see that a 50-question test with all the scores between 35 and 50 must have about 35 really soft, easy questions. Does that mean that a teacher should throw out all the questions that nearly everyone can answer? No, probably not, for the same reason that musician plays a few scales before performing or that a broad jumper takes some practice hops before she takes the big jump. But, once the warmup is over and the real event (that means the test) has begun, it is more efficient, humane, and equitable to ask those questions most likely to yield the widest spread of scors over the available range. The teacher who introduces harder tests will have a little adjusting to do with respect to grade assignment. Obviously, if the average score is about 55%, the lowest passing score will have to be somewhat below 50%. This outcome will bother some people, though it shouldn't. After all, the easy questions just weren't asked, and it is reasonable to assume that most students would have answered them correctly. Then nobody has to worry about getting a headache. --------------------------------------------------- Scores of 100 students on an easy version of a 50-question test. 5 F's 12 D's 47 C's 24 B's 12 A's ******* **** ********** ***** * ********** ***** ** ***** ********** ***** ***** ***** ***** ********** ***** ***** 25 30 35 40 45 50 --------------------------------------------------- Scores of 100 students on a hard version of a 50-question test. 5 F's 12 D's 47 C's 24 B's 12 A's ******* **** ********** ***** ** ********** ***** ** ***** ********** ***** ***** ***** ***** ********** ***** ***** 25 30 35 40 45 50 --------------------------------------------------- For more information, contact Robert B. Frary, Director of Measurement and Research Services 2096 Derring Hall Virginia Polytechnic Institute and State University Blacksburg, VA 24060 703/231-5413 (voice) frary#064;vtvm1.cc.vt.edu ###