Alternative approaches to assessing student engagement rates. Chapman, Elaine

Recent years have seen a revival of interest in the mechanisms by which students’ affective responses to learning tasks moderate knowledge acquisition and skill development (e.g., Wigfield, 1997). Given the emphasis placed on levels of academic achievement in schools, the way in which students acquire knowledge through the learning process has become a primary concern. Several studies have subsequently highlighted the significant role that such factors can play in the learning process (e.g., Mathewson, 1994), laying particular emphasis on those associated with student engagement levels.

The terms school or task engagement are often used to refer to such affective responses. While several lines of inquiry have now converged on the conclusion that these factors play a key role in student learning, findings vary considerably due to differences in definitions and approaches to assessing student engagement levels. The purposes of this overview are to (i) outline some key dimensions of student engagement based on an integrated review of relevant literature, and (ii) describe the various methods that have been used to assess engagement levels in empirical research studies. Specifically, the primary goal was to review approaches to assessing student engagement levels on a classwide basis, and to provide educators with a range of options for developing relevant assessment protocols within their own contexts.

As noted, various operationalizations of student engagement have appeared in published evaluations. Early studies often made use of time-based indices (e.g., time-on-task) in assessing student engagement rates (e.g., Fisher, et al., 1980; McIntyre, et al., 1983; Brophy, 1983). More recently, however, at least two distinct definitions have appeared in the literature (Nystrand & Gamoran, 1992). In the first, student engagement has been used to depict students’ willingness to participate in routine school activities, such as attending classes, submitting required work, and following teachers’ directions in class. For example, Natriello (1984) defined student engagement as “participating in the activities offered as part of the school program” (p.14). Negative indicators of engagement in this study included unexcused absences from classes, cheating on tests, and damaging school property. In this overview, this form of engagement will be referred to as “school process engagement”. Defined in this way, school engagement overlaps considerably with compliance, which in its more general form involves meeting expectations implicit in school contexts.

The second definition used focuses on more subtle cognitive, behavioural, and affective indicators of student engagement in specific learning tasks. This orientation is reflected well in the definition offered by Skinner & Belmont (1993):

Engagement versus disaffection in school refers to the intensity and emotional quality of children’s involvement in initiating and carrying out learning activities…Children who are engaged show sustained behavioural involvement in learning activities accompanied by a positive emotional tone. They select tasks at the border of their competencies, initiate action when given the opportunity, and exert intense effort and concentration in the implementation of learning tasks; they show generally positive emotions during ongoing action, including enthusiasm, optimism, curiosity, and interest. The opposite of engagement is disaffection. Disaffected children are passive, do not try hard, and give up easily in the face of challenges…[they can] be bored, depressed, anxious, or even angry about their presence in the classroom; they can be withdrawn from learning opportunities or even rebellious towards teachers and classmates. (p. 572.)

From a different perspective, Pintrich and colleagues (e.g., Pintrich & De Groot, 1990; Pintrich & Schrauben, 1992) associated engagement levels with students’ use of cognitive, meta-cognitive and self-regulatory strategies to monitor and guide their learning processes. In this view, student engagement is viewed as motivated behaviour that can be indexed by the kinds of cognitive strategies students choose to use (e.g., simple or “surface” processing strategies such as rehearsal versus “deeper” processing strategies such as elaboration), and by their willingness to persist with difficult tasks by regulating their own learning behaviour. In this overview, the term “learning task engagement” will be used to refer to students’ cognitive investment, active participation, and emotional engagement with specific learning tasks. This definition implies the use of three interrelated criteria to assess student engagement levels:

Self-report measures have been used by many researchers to assess the behavioural, cognitive, and affective aspects of task engagement. Items relating to the cognitive aspects of engagement often ask students to report on factors such as their attention versus distraction during class, the mental effort they expend on these tasks (e.g., to integrate new concepts with previous knowledge), and task persistence (e.g., reactions to perceived failures to comprehend the course material). Students can also be asked to report on their response levels during class time (e.g., making verbal responses within group discussions, looking for distractions and engaging in non-academic social interaction) as an index of behavioural task enagagement. Affective engagement questions typically ask students to rate their interest in and emotional reactions to learning tasks on indices such as choice of activities (e.g., selection of more versus less challenging tasks), the desire to know more about particular topics, and feelings of stimulation or excitement in beginning new projects.

A variety of self-report questionnaires have been used in research on student engagement, reflecting the multi-faceted nature of the construct. In a discussion of the key dimensions underlying student reading engagement, Wigfield (1997) suggested that high levels of task engagement were often reflected in factors such as students’ learning beliefs and expectations (e.g., Miller, et al, 1996), self-efficacy (Pintrich & Schrauben, 1992), task interest levels (Schiefele, 1995), and use of effective and/or deep, rather than “shallow” or “surface” learning strategies (Meece, Blumenfield, & Hoyle, 1988). Researchers have used different combinations of these indicators in empirical evaluations. Thus, typical assessment protocols comprise a number of separate indices for assessing the cognitive, affective or behavioural manifestations of task-related engagement. This reflects the fact that no one instrument is likely to be able to comprehensively assess student engagement on all of the construct dimensions listed. Using separate indices also allows educators to adapt the focus of their protocols more towards their own instructional goals.

Attitudes towards, and interests in, learning tasks are highly interrelated constructs and thus often assessed within the same scale. In general, an attitude is defined as a favourable or unfavourable disposition toward specific social objects (Olson & Zanna, 1993). On the other hand, at least two forms of task interest, have been identified (Schiefele, 1991; Krapp, Hidi, & Renninger, 1992). Individual task interests refer to relatively stable and enduring feelings about different activities. Situational interests, in contrast, tend to be more activity- or context-specific. In this view, individual interests are similar to the constructs of attitudes and intrinsic motivation (Wigfield, 1997). Established scales for assessing attitudes and individual/situational task interests are available in most subject areas (see Educational Testing Service, 1992a, 1992b, 1992c). Some researchers have also devised scales that can be adapted for use within any subject area (e.g., Nyberg & Strand, 1979).

Students’ cognitive investment in learning tasks has also been used to index engagement in several studies. For example, the engagement measure used by Meece, Blumefield, and Hoyle (1988) asked students to report on their own use of cognitive, meta-cognitive, and shallow learning strategies in confronting learning tasks. Use of cognitive and meta-cognitive strategies (e.g., I went back over things I didn’t understand”, and “I tried to figure out how today’s work fit with what I had learned before”) was taken to indicate active task engagement, while use of shallow strategies (e.g., “I skipped the hard parts”) was taken to indicate superficial engagement. Similar items were used by Miller et al. (1996) to assess students’ use of deep and shallow learning strategies. The Miller et al. cognitive engagement questionnaire also incorporated separate indices of students’ task persistence and effort. These items were used to assess how students responded to difficult learning problems, and the level of effort they expended on these tasks (e.g., “probably as much effort/the least amount of effort I’ve ever put into a class”).

In addition to asking the question of whether students are engaged in learning tasks, self-report measures can provide some indication of why this is the case. Research into achievement goal orientations, for example, has indicated positive relationships between task or mastery goals, which reflect a desire for knowledge or skill acquisition, and students’ use of effective learning strategies (e.g., Covington, 2000). Several published scales are available for assessing students’ goal orientations, such as the Patterns of Adaptive Learning Survey (PALS) developed by Midgely et al. (2000). Studies have also demonstrated positive relationships between students’ perceived learning control and adaptive learning processes (e.g., Strickland, 1989; Thompson et al., 1998). Several general measures of perceived control are available (e.g., Skinner et al., 1990; Thompson, et al., 1998). Finally, engagement levels have been found to relate positively to students’ confidence and self-efficacy for achieving specific learning outcomes (Schunk & Zimmerman, 1994). Standardized measures are available in a small number of specific subject areas (e.g., Kranzler & Pajares, 1997), while Bandura (2001) provides guidelines for educators to construct their own self-efficacy measures in specific contexts.

In addition to student self-report measures, a few studies have used summative rating scales to measure student engagement levels. For example, the teacher report scales used by Skinner & Belmont (1993) and Skinner, Wellborn, & Connell (1990) asked teachers to assess their students’ willingness to participate in school tasks (i.e., effort, attention, and persistence during the initiation and execution of learning activities, such as “When faced with a difficult problem this student doesn’t try”), as well as their emotional reactions to these tasks (i.e., interest versus boredom, happiness versus sadness, anxiety and anger, such as “When in class, this student seems happy”). The Teacher Questionnaire on Student Motivation to Read developed by Sweet, Guthrie, & Ng (1996) also asks teachers to report on factors relating to student engagement rates, such as activities (e.g., enjoys reading about favourite activities), autonomy (e.g., knows how to choose a book he or she would want to read), and individual factors (e.g., is easily distracted while reading).

Although self-report scales are widely used, the validity of the data yielded by these measures will vary considerably with students’ abilities to accurately assess their own cognitions, behaviours, and affective responses (Assor & Connell, 1992). As such, direct observations are often used to confirm students’ reported levels of engagement in learning tasks. Again, a number of established protocols are available in this area (e.g., Ellett & Chauvin, 1991; Ysseldyke & Christenson, 1993; Greenwood & Delquadri, 1988). While the definitions used in these models vary, most use fairly broad indices to assess engagement. The CISSAR (Code for Instructional Structure and Student Academic Response: Greenwood & Delquadri, 1988), for example, defines engagement in term of behaviours such as attending (e.g., reading from the blackboard), working (e.g., reading aloud/silently), and resource management (e.g., looking for materials).

Regardless of the specific definition of task engagement used, most of these observational studies have used some form of momentary time sampling system. In these methods, the observer records whether a behaviour was present or absent at the moment that the time interval ends. Effective use of this system relies on some form of cuing device to momentarily observe students’ behaviour at pre-specified intervals (e.g., every 10 seconds). Using this method, students’ behaviours are coded as engaged/disengaged at the specific moment in which they were observed. An alternative approach is to use whole-interval sampling, in which students are observed for the full specified time interval (e.g., 10 seconds). In this procedure, a student’s behaviour is scored positively only if the behaviour is exhibited for the full duration of the time interval. While this procedure will produce relatively conservative estimates of student engagement rates, it is also likely to be more sensitive to variations in the consistency and persistence of students’ behaviour.

In classwide observations, approximately 5 minutes of observational data can generally be collected on each target student per lesson. Thus, a 30-minute observation period would allow observations of approximately 5 target students, with 6-7 sessions being required to observe a full class. In addition, to obtain a representative sample of students’ behaviour over the full course of a lesson, observations are generally rotated across students so that each student is observed continuously for only one minute at a time. For example, assuming that 5 students have been randomly selected for observation during a 40-minute lesson (of which only 30 minutes will be observed, allowing for transition time) and using a 10-second whole interval schedule (with 2 seconds recording time), the first target student would be observed 5 times (i.e., over five 10-second intervals) within the first observation minute. After this minute, the observer would move to the next target student and follow the same procedure, rotating their observations across students until each has been observed for a full 5-minute period.

To confirm that measures are standardised across observers, interobserver agreement should be estimated in a pilot run to ensure that observers agree on their interpretation of task engagement. To calculate these estimates, it is necessary for two observers to observe the same target students over the same observational period and then directly compare their ratings in each time interval. A percentage agreement score can be calculated from the number of intervals in which the ratings agreed divided by the total number of intervals observed (in general, 90-100% agreement should be indicated before proceeding).

In addition to the self-report measures described, some educators have used work samples to assess levels of learning task engagement, focusing again on students’ use of higher cognitive or metacognitive strategies in confronting learning tasks. Evidence of higher-order problem-solving and metacognitive learning strategies can be gathered from sources such as student projects, portfolios, performances, exhibitions, and learning journals or logs (e.g., Royer, Cisero, & Carlo, 1993; Wolf, et al., 1990). Hart (1994) provides a comprehensive account of various authentic and performance-based assessment approaches. The efficacy of these methods hinges on the use of suitably structured tasks and scoring rubrics. A rubric establishes a set of explicit criteria by which a work will be judged (Radford, Ramsey, & Deese, 1995). For example, a rubric to assess the application of higher-order thinking skills in a student portfolio might include criteria for evidence of problem-solving, planning, and self-evaluation in the work. A number of formal and informal protocols for assessing students’ self-regulated learning strategies also incorporate components that focus on metacognitive skills (e.g., Pintrich & DeGroot, 1990; Ward & Traweek, 1993; Zimmerman & Pons, 1986). The Metacognitive Knowledge Monitoring Assessment (Tobias, Everson, & Laitusis, 1999) and the Assessment of Cognitive Monitoring Effectiveness (Osborne, 2000) are more targeted measures that are suitable for use in classroom situations. Both instruments have also demonstrated sound psychometric properties in empirical evaluations (Osborne, 2001).

When the focus of an investigation is restricted to a small group of target students, it is often more useful to collect detailed descriptive accounts of engagement rates. Case studies allow researchers to address questions of student engagement inductively by recording details about students in interaction with other people and objects within classrooms. These accounts should describe both students’ behaviours and the classroom contexts in which they occur. This might include, for example, the behaviour of peers, direct antecedents to the target student’s behaviours (e.g., teacher directions), as well as the student’s response and the observed consequences of that response (e.g., reactions from teachers or peers). Case studies generally attempt to place observations of engagement within the total context of the classroom and/or school, and are concerned as much with the processes associated with engagement as they are in depicting engagement levels. Lincoln and Guba (1985) suggest several types of observations recording methods that may be used in case studies (e.g., field notes, context‑maps, and sketches).

This paper provides a broad overview of methods used in assessing learning task engagement on a classwide basis. The paper was designed to provide options for teachers who wish to develop relevant assessment protocols that incorporate a combination of indices across the cognitive, affective, and behavioural domains. In addition to these data, a comprehensive protocol may include measures that address the question of why students do, or do not, engage with particular types of tasks. The latter information can greatly facilitate the interpretation of the overall level indices. Within each of these domain areas, using a range of methods can also strengthen the validity of findings and provide alternative perspectives on the results. Clearly, however, final decisions on protocol components must also take into account any practical constraints within the given context.

The work in this review was supported by a grant from the Lucent Technologies Foundation K-16 Grants Program, to explore the use of student learning teams within ICT-enhanced classroom environments. My sincerest thanks also to Professor Stephen Houghton for his invaluable comments on the revised version of this manuscript.

Aiken, L.R. (1997). Questionnaires and inventories: Surveying opinions and assessing personality. NY: John Wiley and Sons.

Assor, A., & Connell, J.P. (1992). The validity of students’ self-reports as measures of performance-affecting self-appraisals. In D.H. Schunk & J.Meece (Eds.), Student Perceptions in the Classroom (pp. 25-46). Hillsdale, NJ: Lawrence Erlbaum.

Bangert-Drowns, R.L. and Pyke, C. (2001). A taxonomy of student engagement with educational software: An exploration of literate thinking with electronic text. Journal of Educational Computing Research, 24(3), 213-34.

Brophy, J. (1983). Conceptualizing student motivation. Educational Psychologist, 18, 200-215.

Covington, M. (2000). Goal theory, motivation, and school achievement: an integrative review. Annual Review of Psychology, 51, 171-200.

Educational Testing Service (1992a). Attitudes toward school and school adjustment, preschool-grade 3: Annotated bibliography of tests. Eric Document No. 369810.

Educational Testing Service (1992b). Attitudes toward school and school adjustment, grades 4-6: Annotated bibliography of tests. Eric Document No. 369811.

Educational Testing Service (1992c). Attitudes toward school and school adjustment, grades 7 and above: Annotated bibliography of tests. Eric Document No. 369812.

Ellett, C.D., Chauvin, E. (1991). Development, validity, and reliability of a new generation of assessments of effective teaching and learning: Future directions for the study of learning environments. Journal of Classroom Interaction, 26(2), 25-36.

Fisher, C., Berliner, D., Filby, N., Marliave, R., Cahen, L., & Dishaw, M. (1980). Teaching behaviours, academic learning time, and student achievement: An overview. In C. Denham & A. Lieberman (Eds.), Time to Learn. Washington, D.C.: National Institute of Education.

Greenwood, C.R., & Delquadri, J. (1988). Code for Instructional Structure and Student Academic Response (CISSAR). In M.Hersen and A.S. Bellack (Eds.), Dictionary of Behavioural Assessment (pp. 120-122). NY: Pergamon.

Wigfield, A. (1997). Reading engagement: a rationale for theory and teaching. In J.T.Guthrie and A.Wigfield (Eds.), Reading Engagement: Motivating Readers Through Integrated Instruction. Newark, DE: International Reading Association.

Hart, D (1994). Authentic assessment: a handbook for educators. Menlo Park, Calif. : Addison-Wesley Pub. Co.

Herman, K.C., & Tucker, C.M. (2000). Engagement in learning and academic success among at-risk Latino-American students. Journal of Research and Development in Education, 33(3), 129-36.

Kranzler, J., & H. Pajares, F. (1997). An exploratory factor analysis of the Mathematics Self-Efficacy Scale Revised (MSES-R). Measurement and Evaluation in Counseling and Development, 29(4), 215-28.

Krapp, A., Hidi, S., & Renninger, K.A. (1992). Interest, learning, and development. In K.A. Renninger, S.Hidi, & A. Krapp (Eds.), The role of interest in learning and development (pp. 3-26). Hillsdale, NJ: Lawrence Erlbaum.

Marjoribanks, K. (1992). The predictive validity of an attitudes to school scale in relation to children’s academic achievement. Educational and Psychological Measurement, 52, 945-949.

Marks, H.M. (2000). Student Engagement in Instructional Activity: Patterns in the Elementary, Middle, and High School Years. American Educational Research Journal, 37(1), 153-84.

Mathewson, G.C. (1994). Model of attitude influence upon reading and learning to read. In R.B. Ruddell & H. Singer (Eds.), Theoretical models and processes of reading (3rd Edition, pp. 1131-1161). Newark, DE: International Reading Association.

McGarity, J.R., & Butts, D.P. (1984). The relationship among teacher classroom management behavior, student engagement, and student achievement in middle and high school science students of varying aptitude. Journal of Research in Science Teaching, 21(1), 55-61.

McIntyre, D.J., Copenhaver, R.W., Byrd, D.M., & Norris, W.R. (1983). A study of engaged student behaviour within classroom activities during mathematics class. Journal of Educational Research, 77(1), 55-59.

Meece, J.L., Blumenfield, P.C., & Hoyle, R.H. (1988). Students’ goal orientations and cognitive engagement in classroom activities. Journal of Educational Psychology, 80(4), 514-523.

Miller, R.B., Greene, B.A., Montalvo, G.P., Ravindran, B., & Nichols, J.D. (1996). Engagement in academic work: The role of learning goals, future consequences, pleasing others, and perceived ability. Contemporary Educational Psychology, 21, 388-422.

Natriello, G. (1984). Problems in the evaluation of students and student disengagement from secondary schools. Journal of Research and Development in Education, 17, 14-24.

Nystrand, M., & Gamoran, A. (1992). Instructional discourse and student engagement. In D.H. Schunk & J.Meece (Eds.), Student Perceptions in the Classroom (pp. 149-179). Hillsdale>, NJ: Lawrence Erlbaum.

Olson, J.M., & Zanna, M.P. (1993). Attitudes and attitude change. Annual Review of Psychology, 44, 117-54.

Osborne, J. (2000). Assessing metacognition in the classroom: the assessment of cognition monitoring effectiveness. Unpublished manuscript, The Department of Educational Psychology, University of Oklahoma.

Osborne, J. (2001). Assessing metacognition in the classroom: the assessment of cognition monitoring effectiveness. Unpublished manuscript, The Department of Educational Psychology, University of Oklahoma.

Pintrich, P.R., & De Groot, E.V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82(1), 33-40.

Pintrich, P.R., & Schrauben, B. (1992). Students’ motivational beliefs and their cognitive engagement in clasroom academic tasks. In D.H. Schunk & J.Meece (Eds.), Student Perceptions in the Classroom (pp. 149-179). Hillsdale, NJ: Lawrence Erlbaum.

Pintrich, P.R., Smith, D.A., Garcia, R., & McKeachie, W.J. (1993). Reliability and predictive validity of the Motivated Strategies for Learning Questionnaire. Educational and Psychological Measurement, 53, 801-813.

Radford, D.L., Ramsey, L.L., & Deese, W.C. (1995). Demonstration assessment: Measuring conceptual undersanding and critical thinking with rubrics. The Science Teacher, 62(7), 52-55.

Royer, J.M., Cisero, C.A., & Carlo, M.S. (1993). Techniques and procedures for assessing cognitive skills. Review of Educational Research, 63(2), 201-243.

Schiefele, U. (1991). Interest, learning, and motivation. Educational Psychologist, 26, 299-323.

Schraw, G., & Dennison, G. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19, 460-475.

Schunk, D.H., & Zimmerman. B.J. (Eds.) (1994). Self-regulation of learning and performance : issues and educational applications. Hillsdale, N.J: L. Erlbaum Associates.

Skinner & Belmont (1993). Motivation in the classroom: Reciprocal effects of teacher behaviour and student engagement across the school year. Journal of Educational Psychology, 85(4), 571-581.

Skinner, E.A., Wellborn, J.G., & Connell, J.P. (1990). What it takes to do well in school and whether I’ve got it: A process model of perceived control and children’s engagement and achievement in school. Journal of Educational Psychology, 82(1), 22-32.

Smith, M., Duda, J., Allen, J., & Hall, H. (2002). Contemporary measure of approach and avoidance goal orientations: Similarities and differences. British Journal of Educational Technology, 72, 155-190.

Stallings, J., Johnson, R., & Goodman, J. (1985). Engaged rates: Does grade level make a difference? Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, Illinois, April, 1985. (Eric Document No. 267 018.)

Strickland, Bonnie R. (1989). Internal-External Control Expectancies: From Contingency to Creativity. American Psychologist, 44 (1), 1-12.

Sweet, A.P., Guthrie, J.T., & Ng, M. (1996). Teacher perceptions and students’ motivation to read (Reading Research Report No. 69). Athens, GA: National Reading Research Center.

Thompson, M., Kaslow, N.J., Weiss, B., & Nolen-Hoeksema, S. (1998). Children’s attributional style questionnaire – revised: Psychometric examination. Psychological Assessment, 10(2), 166-170.

Tobias, S., Everson, H.T., & Laitusis, V. (1999). Towards a performance based measure of metacognitive knowledge monitoring: Relationships with self-reports and behavior ratings. Eric Document No. 432590.

Tobin, K. (1986). Validating teacher performance measures against student engagement and achievement in middle school science. Science Education, 70(5), 539-547.

Wang, M.C., & Walberg, H.J. (1983). Adaptive instruction and classroom time. American Educational Research Journal, 4, 601-626.

Ward, L., & Traweek, D. (1993). Application of a metacognitive strategy to assessment, intervention, and consultation: A think-aloud technique. Journal of School Psychology, 31, 469-485.

Waxman, H.C., & Huang, S.L. (1997). An observational study of technology integration in urban elementary and middle schools. Journal of Educational Technology Systems, 25(1), 67-77.

Weinstein, C.E., & Palmer, D.R (1990). Learning and Study Strategies Inventory – High School Version. Florida: H & H Publishing Co.

Wolf, D., Bixby, J., Glenn, J., & Gardner, H. (1990). To use their minds well: Investigating new forms of student assessment. Review of Research in Education, 17, 31-74.

Ysseldyke, J., & Christenson, S. (1993). TIES-II: The Instructional Environment System-II. A system to identify a student's instructional needs. Eric Document No. 367072.

Zimmerman, B.J., & Pons, M.M. (1986). Development of a structured interview for assessing student use of self-regulated learning strategies. American Educational Research Journal, 23, 614-628.

Elaine S. Chapman (B.A.Hons, Ph.D.) is a Lecturer in the Graduate School of Education at the University of Western Australia, where she teaches and conducts research in the areas of Research Methodology and Educational Assessment.

Elaine Chapman
The Graduate School of Education
The University of Western Australia
Nedlands Perth Western Australia. 6009.