ALL OF THE ABOVE/NONE OF THE ABOVE. Twelve Par-L Subscribers Respond, May 1-7, 1995. On Mon, 1 May 1995, Par-L received a note from the Listmanager (Neil Sapper): Par-L has been very quiet for several months. I feel guilty in using the resources of the host of the list. In the interest of starting a virtual conversation, let me try this ice-breaker as a question of the week. ------------------------------------------ Prof. I. M. Disorganized has decided to use test generating software. The program will permit the user to create up to four forms of the same test (Forms A, B, C, and D) by scrambling the items on the forms. Item 1 on Form A appears as Item 4 on Form B, Item 3 on Form C, and so on. Several (all?) of the items on Prof. Disorganized's test are multiple choice items. Within these items, the user can scramble the M/C answer choices. Choice A. in Item 1 on Form A appears as Choice C in Item 4 on Form B, and so on. The problem, for Prof. Disorganized, is not in the scrambling. The test generator does that for the user. The difficulty is that Prof. Disorganized has used the fiendish "all of the above" and "none of the above" choices in several M/C items. If the answer choices are scrambled, there is potentially nonsensical situation where "all of the above" or "none of the above" will appear as Choice A. Or the AOTA (all of the above) or NOTA (none of the above) choices are in reverse order. What is poor Prof. Disorganized going to do about an AOTA choice appearing as Choice A? The operators are standing by. -NS ----------------------------------------- On Mon, 1 May 1995 Howard Mount (Accounting, Seattle Pacific University) replied: Dear Prof. Disorganized: I could do some double-checking, but I know I have used some low cost (free with adoption of text) test generator which marks NOTA and AOTA an does not include them in the scramble, i.e., they remain the last items. If someone is really interested in the specific software, I'll do a search of my software cupboard! Howard E. Mount ----------------------------------------- Also on Mon, 1 May 1995, Tom Donlon (Test Development and Research, Edison State College) wrote: Dear Prof. Disorganized: Well, here's a top of the head example: I. For many test takers there is a psychologically correct order for the options they are to consider. II. Dropping an "all of the above " or an "all of the options" option in the middle of more substantive, fact-oriented options is often cognitively stressful for test takers. III. It's really fun to confuse the test taker with complex information fields that foster misinterpretation and confusion. Which of the assertions above does Tom Donlon believe are correct? A. I only B. II only C. III only D. I and II only E. I, II, and III My answer is D. My premise is that you should have a kind of hierarchical arrangement to the answers, from simple to complex, and that a shuffling makes the material harder. Then there's things like data sufficiency, which I will not crowd in here. Tom ------------------------------------------------- Then on Tue, 2 May 1995 Gary Fish (Maine Board of Pesticides Control) wrote: Dear I.M. The answer is to not use all of the above or none of the above or if you absolutely can't get away from it, use "All of these" or "None of these". Gary Fish ------------------------------------------------ On Tue, 02 May 95, Joe Moore (Business, Arkansas Tech University) added: Dear Prof. Disorganized: I would suggest not to use the AOTA or NOTA choices. Theseforms of responses do not tell the Prof. anything about the studentsknowledge of the subject - except that they may have great test smarts or the student knows what does not answer the question. If the purposeof the test is to measure knowledge, then the questions need to be written so that the student may display what they have learned, as well as the instructor being able to give an objective grade. In brief, rewrite the questions so that there is one correct answer, or allow the students to choose more than one answer as correct without using the AOTA choice. Joe Moore ------------------------------------------------------ Late Tuesday, May 3, Albert F. C. Wehlburg (Theater, U. Of Florida) offered a response to the query of the week: To Dr. Disorganized; I used to use the ALL OF THE ABOVE as a choice for the answers. So now I changed that to ALL OF THE ANSWERS PROVIDED, or NONE OF THE ANSWERS PROVIDED. So now if I scramble then answers it still works OK. Students are used to have these options at the bottom of the list provided. But there is no law which requires that they be at the bottom of the list. Albert F. C. Wehlburg -------------------------------------------- Lois Frankel (?) joined the thread about M/C answer choices on Wednesday, may 3, 1995, in response to Albert Wehlburg's solution to the scrambling of AOTA/NOTA statements in an earlier post to par-l: Dear Prof. Disorganized, "None of the answers provided" can never be correct, since choosing it indicates that exactly one answer (itself) is correct. "None of the above" or "all of the above" do not have that problem, not being self-referential of the above. Lois Frankel ---------------------------------------------- Dave Eddings (Computer Coordinator--Center of Emphasis, Columbia State Community College) spoke thoughtfully about the AOTA/NOTA propreity issue on Wednesday, May 3: Dear Prof. Disorganized, As a purely academic evaluation, the distractors may be considered inappropriate in a test. Please remember, however, that you may have to modify your opinion slightly to compensate for the fact that the software is going to make the selections meaningless after they are scrambled unless you make some accomodations. If you were writing out the tests one version at a time, you could change these accordingly. If you are going to take advantage of ParTEST's multiple test version capability, you're going to have to adapt accordingly. Dave Eddings ------------------------------------------ Late Wednesday, May 3, the Par-L manager, Neil Sapper (History, Amarillo College) wrote: Professor I. M. Disorganized (students call him "Ol' Diz" or "Dizzy" outside of class) mentioned a helpful source for test item writers: _"All of the above...: A Guide to Classroom Testing and Evaluation"_, 2nd ed. by Kenneth K. Ortiz and Ralph Lewis (1988). In Chapter 4 (Writing Multiple-Choice Items, pp. 28-29), Ortiz and Lewis offer RULE #4: AVOID THE USE OF "ALL OF THE ABOVE." Using this type of alternative makes it possible to answer the item on the basis of partial information. Because the student is to select only one answer, he can detect "all of the above" as the correct choice by simply noting that two of the alternatives are correct. He can also detect it as a wrong answer by recognizing that one of the alternatives is incorrect. FAULTY EXAMPLE: What statistic is defined as a measure of dispersion? A. Standard deviation B. Range C. Variance D. All of the above IMPROVED EXAMPLE: What statistic is defined as a measure of dispersion? A. Median B. Variance C. Mode D. Mean RULE #5: "NONE OF THE ABOVE" SHOULD NOT BE USED. It is not possible to use this type of alternative with the best-answer type of item. It is most widely used with computational problems so that the student will not be able to estimate the answer without doing the entire computation. FAULTY EXAMPLE: Which of the following processes are NOT found in humans? A. Digestion B. Respiration C. Photosynthesis D. None of these. IMPROVED EXAMPLE: Which of the following processes are NOT found in humans? A. Digestion B. Respiration C. Photosynthesis D. Circulation What does par-l think of these positions on the AOTA/NOTA issue in test items? -NS In response to Rules 4 and 5 on M/C answer choices, John R. Hills (Educational Research, Florida State U.) wrote on Wed, May 3, 1995: Dear Prof. Disorganized, "None of the above" is an excellent alternative to use in any item requiring calculation. In fact, the best alternatives for such items are "answers" one gets from doing the calculations incorrectly in the ways learners most often make mistakes. As some one pointed out, NOTA keeps the examinees from concluding that if he does not come up with one the the alternatives that is listed, he must have done the calculation incorrectly. To function properly, of course, NOTA must be the correct answer at least occasionally. Rely on your item analysis results to inform you whether AOTA and NOTA are effective distracters. If not, do not continue to use them. And never use them just because you think you want a certain number of alternatives and can't think of anything else. Of course, do not use NOTA in a question expressed as a negative because that results in a double negative. That is the most serious problem with the example for "Rule #5" in which the example's stem is, "Which of the following processes are NOT found in humans?" NOTA would have to mean that, "None of the above are NOT found in humans." No one should use test items so awkwardly phrased. AOTA is of limited usefulness for the reasons mentioned in other postings. However, once in a while--such as when you want to see if students recognize all of the assumptions-- an item using AOTA might prove useful. It is, of course, the keyed answer for such an item, and AOTA must be used elsewhere as a distracter or students will soon catch on that it is the keyed answer where ever it appears. Item writing requires that you employ your intelligence; it cannot be reduced to a mechanical procedure. It does take time. John R. Hills --------------------------------------------- On Wednesday, May 3, 1995 Howard Mount (Accounting, Seattle Pacific U.) responded to John R. Hills: Dear Prof. Disorganized, In response to John R. Hills, the best use of NOTA that I've seen is one that says: d. None of the above. The answer is____________. This, as indicated is best used for a calculation type question. It slows down the grading process, but gives the student a lot more to think about. I think another aspect of multiple choice that I have found useful (I teach accounting classes where there's more "calculation"), is to give partial credit for some of the distractors. This requires that I choose distractors that have a particular error that the student is likely to make. This has taken a lot of the fire out of students who argue that multiple choice questions are all or nothing while more subjective testing gives partial credit. I, too, give partial credit. It would be nice if it were possible in a test generator/administrator to be able to put either points or weights on individual choices. Howard Mount ------------------------------------ Also on Wednesday, May 3, 1995 Dennis Roberts (Educational Psychology, Pennsylvania State University) responded to John Hills: Dear Prof. Disorganized, John may disagree with this but ... NOTA may be helpful but may not be either. For problem solutions ... if one does a problem ... and comes up with an answer that is NOT one of the choices but, happens to be wrong ... the NOTA choice will be selected since it fits with what the respondent got. In this case ... it does not really make the examinee go back to find the error of his/her ways ... it's a legitimate response. Also ... what if NOTA is the correct choice but, the examinee does the calculation wrong ... but it matches one of the distractors. Here again ... there is a match ... and no particular reason why the examinee shuold go back and try again. The POTENTIAL advantage of NOTA choices ... as John said when they are the correct ones now and then ... is to force the student to now and then recheck his or her work. But ... in the two cases above ... I fail to see how that will happen. So... when WILL it be helpful? I am Dennis Roberts ------------------------------------ On Wed, 03 May 1995 13:26:39 Howard L. Yarbrough" (Biology, Chattanooga State Community College) took time out from the last week of the spring term: Dear Prof. Disorganized, On the advisability of NOTA/AOTA items, just a comment. For the past 10 years I have served on a committee for a national licensing exam for teachers of biology. The organization responsible for the exam (which I can not disclose due to their requirement (but I'm sure you all know of whom I speak) long ago prohibited us to use AOTA/NOTA when constructing items. Their detailed statistical analysis indicated that these items failed to provide the discrimination which the exam was attempting to measure. Howard Yarbrough ------------------------------------ On Wed, May 3, 1995, Albert F. C. Wehlburg (Theater, U. Of Florida) replied to Lois Frankel through Prof. Disorganized: Dear Prof. Disorganized, I used to use the ALL OF THE ABOVE as a choice for the answers. SO now I changed that to ALL OF THE ANSWERS PROVIDED, or NONE OF THE ANSWERS PROVIDED. So now if I scramble then answers it still works OK. Students are used to have these options at the bottom of the list provided. But there is no law which requires that they be at the bottom of the list. Lois Frankel wrote that "None of the answers provided" can never be correct, since choosing it indicates that exactly one answer (itself) is correct. "None of the above" or "all of the above" do not have that problem, not being self-referential of the above. Lois is correct. Maybe I will change it to NONE OF THE OTHER ANSWERS PROVIDED. Although my students seem to be able to handle it. Albert F. C. Wehlburg ------------------------------------ On Thursday,May 4, 1995, Don Rueter (Computer Center, Orange Coast College) wrote: Dear Prof. Disorganized, I don't want to argue the efficacy of AOTA's and NOTA's, but I do note on page 69 of the ParTEST version 3.70/4.0 manual that "For multiple-choice questions, answer scrambling occurs only if there are four or more answerchoices to the question and if scrambling is requested. If there is a fifth answer, only the first four answer choices (A-D) are scrambled." Thus if your AOTA or NOTA is choice E, you have no problem. How about another question. Is it possible to save the Printing Instructions so that it isn't necessary to reenter margins and page length each time the program is run? Don Rueter ----------------------------------------- This post concluded the All of the above/None of the above thread on Par-L. - Neil Sapper, Par-L Listmanager.