Learning Outcome Testing: A Statewide Approach to Accountability by John A. Willis [Coordinator, Statistical Student Assessment Services--West Virginia Department of Education, Charleston, WV Originally appearing in _T.H.E. JOURNAL_, August 1988, pp. 69-73. In West Virginia, the state Board of Education has policies concerning monitoring and assessing student progress toward mastery of learning outcomes. The West Virginia Department of Education initiated the Learning Outcome Testing Program (LOTP) to help counties meet the requirements of those board policies. Under the program, the state Board of Education will provide counties with test questions measuring the learning outcomes approved by the board. These questions are designed for the use by teachers in formative evaluations of their own instruction and their students' progress. Teachers will have access to an extensive bank of such questions, from which they will select the particular items they use in the classroom. The program is operated out of the Bureau of General, Special and Professional Education (GSPE) and the Bureau of Vocational, Technical and Adult Education. -A Legal Issue- The LOTP actually had its origins in an important 1981 state judicial decision known as the Recht decision. The lawsuit was filed against the state Department of Education by the mother of two Lincoln County students. The plaintiff claimed the education her children were receiving was not as good as that given in other counties. The case reached the state Supreme Court, where Judge Arthur Recht found that the state Department of Education was not in compliance with the state constitution. Recht's decision directed the department to correct its problems. In response, the department developed the 1983 Master Plan. The plan essentially provided learning outcomes for all the curriculum areas taught in the schools. These became our standards for working with schools and monitoring schools and counties. Somehow, assurances had to be provided that students were learning and had mastered the learning outcomes. Criterion-referenced testing was seen as a way to provide such assurances. Personnel in the bureaus of vocational and general education began planning criterion-referenced testing of state-board-approved learning outcomes. The 1984 state Legislature appropriated $450,000 for a learning outcome testing program. In the Bureau of General, Special and Professional Education, discussion heated up in the Spring of 1984. In June, the assessment staff wrote project plans for an "Outcome- Referenced Testing Program." In October, State Assistant Superintendent and GSPE Bureau Chief Dr. John Pisapia convened a meeting of some of his staff and two staff members from the bureau of vocational education to get a consensus on a testing program. Based on that meeting, a "concept" paper on learning outcome testing was drafted in December of 1984. -Objectives and Assumptions- Pisapia wrote Roy Truby, then state schools superintendent, a memorandum in January 1985 that stated the objectives and four assumptions of the LTOP. The objectives were given by this paragraph: Policy 2510 asserts as part of the program assurance component that county school districts must have a system for monitoring student progress toward the mastery of learning outcomes. The Learning Outcome Testing Program (LOTP) is being designed and initiated to provide data for use in the ongoing instructional process. Hence, the LOTP provides input to the student monitoring system. The following four assumptions were listed: 1. It is important to view the LOTP as part of the overall instructional plan, in which test information is used to adjust the system. 2. It should not be viewed at this time as a mechanism for making final, summational judgement or decisions about individual students, school districts or the state with respect to progress toward mastery of learning outcomes. 3. Since the LOTP is being viewed from an instructional orientation, it is not the intent of the LOTP to generate a single comprehensive test or even a series of tests. The LOTP will generate an extensive pool of valid and reliable items. 4. Although eventually it would be possible to use the pools of items to generate, in each curricular area, tests which could be used for comparisons or judgemental purposes about progress toward mastery of learning outcomes, the LOTP is not currently being perceived in this limited fashion. From the outset, then, the LOTP was conceived as a program to provide test items referenced to the outcomes and for formative evaluation. We so characterized our LOTP efforts at regional meetings in the Spring of 1985 at August's annual state Leaders of Learning Conference. -In the Hands of Every Teacher- In June and July, the staff conducted item-writing workshops in which 74 match teachers, assisted by curricular supervisors and measurement specialists, developed more than 13,000 test items for 19 K-12 content areas. In July and August of 1987, item writing workshops were held for K-12 science. Currently item development is under way in music and reading. We want to put computerized LOTP item banks in the hands of every general-, special- and vocational-education teacher in the state, along with software to enable teachers to easily withdraw items, in the form or tests. Teachers can print these tests with a dot-matrix printer and duplicate them to hand out to a class. After the students take the test, teachers may score them by hand or with a scanner. In either case, additional software will enable teachers to tie together test results, learning outcomes and objectives so they can report what has been taught and tested and who has mastered, partially mastered or not mastered the learning outcomes and objectives. Currently, programs of study with approved learning outcomes include art, business, education, consumer and homemaking studies, diversified cooperative training, driver's education, foreign language, health, health occupations, industrial arts, library/media, marketing education, mathematics, music, occupational home economics, physical education, pre-vocational exploration, reading, science, social studies and vocational agriculture/agribusiness. The LOTP has the potential to affect the 350,000 K-12 students in general, special and vocational education and to impact curriculum alignment, instruction, students' learning and self-esteem, teachers' job satisfaction and county monitoring efforts. Student achievement will be enhanced by continual formative testing of curriculum objectives, which, in turn, are covered by textbooks and supplementary materials. Student self-esteem will also be enhanced as students experience success in being tested with items that correspond to what they've been taught. Much of the busywork in recordkeeping and test-making will be eliminated with the item bank, testing and curriculum-management software we propose. The math and science item banks alone can be considered "benefits" of teaching in West Virginia. Recent studies of teachers' instructional decisions show why they rely heavily on the questions, quizzes and tests in textbooks and on assessments they design and construct themselves rather than on standardized tests or test scores. (NOTE: See for instance, Rich Stiggins, "Improving Assessment Where It Means the Most: In the Classroom," _Educational Leadership_, October 1985.) Curriculum alignment will be furthered by the provision of test questions that both match learning outcomes and can be easily accessed for classroom test development. There will be approximately 125,000 items written or acquired for the subject areas named above that currently have defined learning outcomes. These will be grouped in approximately 120 item banks. Such a volume of items necessitates computerized item banking and test- generation. -Testing in Two Different Environments- The LOTP will provide resources to teachers whereby they can easily test their students and their own instruction, vis-a-vis the learning outcomes, and keep track of students' progress. The scope in which this monitoring will occur ranges from simple classroom testing applications to mastery-learning situations. In simple classroom testing, the teacher tests chiefly to assign grades. LOTP items are seen as substitutes for textbook questions and for teacher-made questions. The classroom environment involves frequent tests, since unit, midterm and final tests occur on a regular basis. There is relatively little need for individually tailored tests or for multiple parallel forms, since students get the same test at the same time. For remediation purposes, LOTP item banks will provide for repeated testing on the same content. Mastery-learning situations are those in which individual students are proceeding through the curriculum at their own pace. This testing scheme allows students to be tested on the same content at different times and will allow for retesting of students who didn't pass a previous test. Thus, the system will allow for many unique tests of the same content. Tests need to be available at the classroom level, development and scoring need to have quick turnaround, each student's place in the curriculum needs to be known, and test administration schedules should be flexible. -Where Is the LOTP Now?- Currently, we have approximately 18,000 items in 19 mathematics and 13 science content areas, K-12. And we have 9,000 items in nine vocational education units. The West Virginia Department of Education has spent well over $250,000 in developing these item banks. To develop valid and reliable LOTP items, the Bureau of General, Special and Professional Education utilizes practicing and experienced teachers from West Virginia as item writers. The bureau has conducted four ten-day workshops at which 125 teachers have been trained in item-writing techniques and have written the math and science items mentioned above. Eight county curriculum supervisors also have been involved in the workshops as content reviewers, checking the curriculum validity of the written items. Another way of obtaining test items that match prescribed learning outcomes and objectives is to contract teams of item writers and psychometricians at higher education institutions to develop them. This is the route being taken by the Bureau of Vocational, Technical and Adult Education. Thus far, the LOTP has enabled more than 100 teachers to be trained in item-writing and teacher-made test construction. The project has developed more than 25,000 test questions designed for the West Virginia school curriculum by teachers and instructors who are able to draw upon their experience, their knowledge of the subjects and their understanding of children. In addition, the LOTP has fostered a sense of ownership in the program among professional educators in the state. -What About the LOTP Software?- More than a year and a half ago, units within the vocational and general education bureaus began investigating computer software for use in the LOTP independently of one another. In September 1986, the teams drew up the criteria for evaluating such programs based on demonstrations from half a dozen vendors, hand-on review of packages, and their own knowledge of what teachers need in their classrooms and laboratories. In October 1986, they evaluated existing packages, but none met the criteria deemed essential by both bureaus. In November 1986, an interdepartmental committee appointed by the bureau chiefs representing general, special and vocational education to acquire software for the LOTP through the formal "request for proposals" (RFP) process of the Information System Service Division of the Department of Finance and Administration. The committee was made up of two members from the vocational bureau and two from general and special education. The RFP specified that software should run on IBM PCs and compatibles. We'd already invested quite a bit in PCs, especially in vocational education. That bureau has established a statewide computer network, and vocational programs also have a number of computer labs based on a Corvus network and IBM PCs. This has influenced general education programs, some of which are using IBM PCs in lab installations. However, because we also have a large number of Apple IIs in schools, the evaluation committee awarded extra points to software that was available in both versions. In fact, we plan to install the item banks onto the statewide network to allow our Apple II users to download the items onto their systems. Four proposals were received, two vendors were chosen to give demonstrations, and the final selection was made in June 1987. We entered into a contract with National Computer Systems for acquisition of ParTEST Software by Economics Research, Inc. of Costa Mesa, California (which is now being distributed by NCS as examSYSTEM Test Generator) and a modified version of NCS's own IMSplus package. ParTEST will be used for the item banking and test-generation component, and IMSplus for the curricular management component. The examSYSTEM (aka ParTEST) test item banking and test- generation software represents the most basic classroom level of a comprehensive instructional management system. As we intended when we began the LOTP, the program allows individual teachers to access learning-outcome-based item banks and generate tests for classroom use across various user-defined categories (i.e., learning outcomes, skills, objectives, tasks, _etc_.). As such, it will allow teachers to use test results, learning outcomes and objectives to better monitor and report student progress. This basic information, generated in the classroom, can then be transferred to other, more comprehensive instructional data management systems. Specifically, the software chosen was recommended for classroom use because of the following characteristics: *It is a teacher-centered, menu-driven program compatible with existing technology currently in the classroom. (The minimum configuration is a 256K, stand-alone computer and dot- matrix printer.) *It affords teachers flexibility in terms of applications (i.e., testing across a number of variables, the ability to accommodate various types of test questions and the ability to customize tests for specific purposes.). *It's simple to use, requiring very little training or computer experience. *It is easy of the user to add items, edit item banks or transfer items into existing banks from standard ASCII files. *The program is able to store item banks on either floppy or hard disks, with the capability of being networked in those schools with appropriate equipment. *It eliminates much of the busywork of test construction and management. *In addition, ParTEST is available in an Apple II version. (NOTE: This is no longer true of any ParSYSTEM software.) The use of this software, as the basic delivery component of a computerized learning outcome testing program, has the potential to positively impact curriculum alignment, instruction, students' learning and self-esteem, teachers' job satisfaction, and county monitoring efforts. Without such an instructional tool, the delivery, management and application of the approximately 18,000 currently developed test items would be greatly hindered. We in the state of West Virginia are looking forward to full-scale implementation of this project, and we hope that our experiences in an endeavor of this magnitude will prove useful and inspiring to other states. -30- Return-path: Received: from ac1.actx.edu by cusrvb.cua.edu (PMDF V4.3-10 #9933) id <01HP00ILI5F48ZF2ND#064;cusrvb.cua.edu>; Thu, 06 Apr 1995 00:59:16 -0500 (EST) Received: from pcad-ml.actx.edu by ac1.actx.edu with SMTP (1.37.109.4/15.6) id AA14163; Wed, 5 Apr 95 17:25:03 -0500 Received: from SAS/MAILQUEUE by PCAD-ML.ACTX.EDU (Mercury 1.13); Wed, 5 Apr 95 17:25:46 GMT-6 Received: from MAILQUEUE by SAS (Mercury 1.13); Wed, 5 Apr 95 17:25:29 GMT-6 Date: Wed, 05 Apr 1995 17:25:20 +0000 From: "Neil G. Sapper" Subject: LOTP.TXT To: rudner#064;CUA.EDU Message-id: <1E6DF6810CB#064;PCAD-ML.ACTX.EDU> Organization: Amarillo College PCAD-ML X-Mailer: PMail v3.0 (R1a) Content-transfer-encoding: 7BIT Priority: normal