Edys Quellmalz, Patricia Schank,
Thomas Hinojosa, and Christine Padilla
SRI International, Center for Technology in Learning

Understanding the central role that performance assessment plays in standards-based reform, educators are seeking ways to use these assessments to test student learning. Education agencies need pools of performance tasks to use in their student assessment programs and in evaluations of state and federally funded programs. Reform projects need standards-based assessment, too, as do teachers who are trying to implement reforms. Experience indicates, however, that the level of effort and costs of developing performance assessment tasks are very high (Quellmalz, 1984).

To meet the need for innovative approaches for sharing exemplary assessment resources, to facilitate the development of new ones, and to further understand how the use of standards-based performance assessment can advance educational reform, SRI International is developing Performance Assessment Links in Science (PALS), an on-line, standards-based, interactive resource bank of science performance assessments. Coupled with the development of the resource bank is a program of research on effective use of these resources.

This digest describes work-in-progress. SRI is currently seeking organizations to participate in implementation studies involving the use of PALS. The "ideal" professional development model would be for an organization or group of schools to want to focus on classroom applications of science performance assessment and to use the PALS resources in an initial professional development institute, followed by several school-year meetings where teachers would bring samples of student work produced in response to PALS investigations, with subsequent work on designing additional classroom science assessments. For the accountability studies, SRI will work with a few assessment programs from states, districts, or specially funded programs that are interested in using the on-line, secure tasks in the Accountability Pool. Assessment programs might use PALS to search, select, and plan assessment administrations and/or conduct on-line rater training and scoring sessions.

Performance Assessment in School Reform

Reform-minded programs have set out to develop alternative forms of student assessment that call for students to construct rather than select responses. Performance assessments are generally valued for testing students' deep understanding of concepts and inquiry strategies, for making students' thinking visible, and for measuring skills in communicating about their science knowledge. In addition, performance assessments can present authentic, real-world problems that help students to show they can apply academic knowledge to practical situations. On the other hand, performance assessments are time-consuming and costly to develop, logistically demanding, and of questionable utility if not developed and scored according to sound measurement methods.

Technology in Assessment Reform

Technology offers a powerful strategy for increasing the ease with which educators can access and use standards-based student assessment. Technology can be used to efficiently archive numerous assessments for ready browsing. Currently, some sets of performance assessment tasks are distributed on CD-ROM, and there are plans to place released test items on the Internet, although these resources are not yet coordinated or easily accessible to programs and schools. Electronic networks can go beyond storage by supporting the growth of a community of colleagues and leveraging expertise, regardless of geographic location and institutional base. Networks can offer templates and guidelines and support collaborative development and on-line conversations about the alignment of tasks with standards, quality of tasks and student work, on-line training, scoring, and standards-setting sessions. Technology can also support simulated investigations and data collection and analysis of student responses. The exponential advances taking place in technology promise to revolutionize assessment practices and education reform (Kozma & Schank, 1998; Quellmalz, 1999).

Technology in Professional Development

New professional development models are needed to provide teachers with greater opportunity to access, discuss, incorporate, and co-construct assessment resources and other reform-based materials (Little, 1993). Current efforts find it difficult to reach many teachers and to maintain discourse, and teachers have difficulty implementing new ideas back at their schools. Technology can help provide mechanisms for teachers to overcome their isolation and make more effective use of their time spent on professional growth.

Performance Assessment Links in Science (PALS)

PALS provides an on-line assessment resource library designed to serve two purposes and user groups: (1) the accountability requirements of state education agencies and specially funded programs, and (2) the professional development needs of classroom teachers. SRI is developing PALS under a grant from the Instructional Materials Development (IMD) program within the National Science Foundation with two primary goals:

(1) To develop a two-tiered on-line performance assessment resource library composed of performance assessment tasks for elementary, middle, and secondary levels. One tier will be a password-protected, secure Accountability Pool of science performance assessments for use by state assessment programs and systemic reforms. The second tier is for use by teachers and professional development organizations. The Professional Development Pool provides performance-based science assessments that have been used successfully in large-scale (state or national) assessment programs and have been released.

(2) To evaluate the effectiveness of policies, implementation models, and technical quality requirements for the use of the two tiers of PALS.

In our design for PALS, experienced assessment programs can contribute standards-based science assessment tasks with documented technical quality to the PALS on-line library. The Accountability Pool will be composed of password-protected, secure tasks accessible only to approved assessment program staff. Assessment programs can thus share their resources and enjoy a much larger pool of science performance assessments to use or adapt for their testing administrations. The PALS resource library can provide large, continually updated collections that can support efficient searching, selection, and printing.

The Professional Development Pool contains resources that are of documented technical quality and have been released for access by teachers and professional development groups. Pre-service and in-service programs, for example, can reach teachers in geographically distributed and remote locales, resulting in great savings of travel and materials expenses. On-line guidelines and templates can support classroom use of science performance assessments. Teachers may administer the performance tasks as part of their classroom assessments, adapt them, or use them as models for developing similar investigations.

To help users design test administration forms that cover important science standards, the on-line system provides assessment planning charts (Stiggins, 1994). PALS automatically produces an assessment planning chart to display coverage of standards by the performance tasks selected by the user.

The science performance assessment library includes the scoring rubrics designed to judge the quality of student responses to a task or set of tasks. To bring meaning to the scoring rubrics, a library of exemplars of scored student work is included.

Rater training materials are not routinely published by assessment programs. SRI has developed specifications for on-line rater training and scoring so that each agency wishing to take advantage of the PALS system can convert traditional, stand-up training procedures and calibration to written form, assemble training packets, and test their effectiveness in on-line delivery.

An essential component of PALS is the documentation of the technical quality of each task in the library. PALS contains the technical quality indicators provided for the field-tested science tasks. Since the resource library will be stocked only with tasks that have survived a systematic development process, the tasks and rubrics will also have been subjected to content validity and bias reviews.

Issues and Expansion of the System

The project will be addressing a number of issues. One will be the procedures for identifying the science standards that tasks are designed to test and the nature of the groups that will make the alignment judgments. Another issue relates to policies for accessing the resources that have been developed with funds from different sources and that represent assessment materials being distributed by various organizations. A third issue relates to the criteria and procedures for reviewing tasks submitted for inclusion in the bank. Finally, the system must develop strategies for operating, maintaining, and expanding the resources.

PALS uses technology to efficiently archive numerous assessments for ready browsing by teachers, and goes beyond mere storage of assessments to support cross-links with standards, tasks, scoring criteria, and annotated student work. These resources can be shared by assessment programs, allowing them to have access to a large pool of performance assessments to use or to adapt for their testing administrations. Teachers can administer the tasks as part of their classroom assessments, adapt them or use them as models for developing their own investigations, and contribute their adaptations to the resource bank for other teachers to use.

As an on-line library alone, PALS is a valuable set of resources. However, the growing body of professional development research and the results of our pilot study suggest many benefits of integrating more collaboration support into PALS. SRI plans to integrate an online meeting/discussion component to help the community members leverage expertise, regardless of geographic location. This component could be used to support collaborative development and on-line conversations about the alignment of tasks with standards, the quality of the assessment tasks and student work, rater training and scoring, and standards setting. The authors believe that, by taking advantage of new models of professional development that include innovative digital technologies, PALS will provide excellent professional development opportunities for teachers.

References

Kozma, R., & Schank, P. (1998). Connecting with the twenty-first century: Technology in support of education reform. In D. Palumbo and C. Dede (Eds.), Association for Supervision and Curriculum Development 1998 yearbook: Learning with technology. Alexandria, Virginia: Association for Supervision and Curriculum Development.

Lieberman, A. (1996, November). Creating intentional learning communities. Educational Leadership, 51-55.

Little, J. (1993). Teachers' professional development in a climate of educational reform. Educational Evaluation and Policy Analysis, 15(2), 129-151.

Quellmalz, E. S. (1984). Designing writing assessments: Balancing fairness, utility, and cost. Educational Evaluation and Policy Analysis, 6, 63-72.

Quellmalz, Edys (1999). The role of technology in advancing performance standards in science and mathematics learning. In Kathleen Comfort (Ed.), Advancing standards for science and mathematics learning: Views from the field. Washington, DC: AAAS.

Stiggins, R. J. (1994). Student-centered classroom assessment. New York: Macmillan College Publishing Company.