|
Scriven, Michael (1999). The nature of evaluation part i: relation to psychology. Practical Assessment, Research & Evaluation, 6(11). Retrieved August 18, 2006 from http://edresearch.org/pare/getvn.asp?v=6&n=11 . This paper has been viewed 18,592 times since 11/13/99.
The Nature of Evaluation Part I: Relation to psychology
Michael Scriven,
Claremont Graduate University
The discipline of evaluation is devoted to the systematic determination of merit,
worth, or significance. It is divided into fields according to the type of entity
evaluated--for example, program evaluation, or personnel evaluation--and there
are more than twenty of these recognized fields of evaluation. Some specific
aspects of evaluation methodology have been developed to solve problems of
evaluation in only one or a few of these fields (e.g., bias control in panel selection,
systematic side-effect identification in program evaluation, road-testing techniques
in product evaluation). However, the underlying logic of the process of
evaluation--for example, the difference between merit and worth, or between
grading and ranking--and a substantial portion of its general methodology (e.g.,
techniques of measurement, causality determination, applying the requirement of
informed consent) are shared across all or many of these fields. Many of these
general techniques (the 'general methodology') come from the applied social
sciences, and are learnt by students in the normal course of education in those
fields. But the logic of evaluation has been developed for and applies only to
evaluation; and the field-specific methodologies of evaluation must also be
mastered in order to deal with evaluation in the fields to which they apply.
Teaching evaluation therefore focuses on these evaluation-specific topics, the
general logic and the special methodologies.
This article addresses the role of evaluation, the basic logic, and a description
of how the field is structured. A separate article describes some of the basic logic
of evaluation skills and methodological skills that need to be mastered..
Evaluation in applied psychology
Just as it was previously found that a good grasp of probability and statistics had
become an essential tool for a great deal of work in applied psychology, so we
now find that a knowledge of the logic of evaluation and of some of its specialized
methodologies are increasingly crucial for much investigation in applied social
science. Funding agencies letting evaluation contracts or assigning staff to
evaluation increasingly want to know not only what is happening when an
intervention is supported, and exactly what causes the results - familiar questions
for applied social science - but also
- whether the intervention is worth what it
cost,
- whether there are unintended bad results as well as planned good ones,
- whether the methods used in the intervention were proper by current
professional and ethical standards, and
- whether there are better ways to do the
same thing.
This is the domain of evaluation, and none of those questions can be
answered reliably without some use of its special logic and its special
methodologies.
The Basic Logic of Evaluation
In order to reach evaluative conclusions it is usually necessary to establish or
identify two kinds of premises:
factual premises (e.g., about nature, performance
or impact) and
value premises (e.g., about the relevant legal or scientific
principles).
There are usually many of these premises in the evaluation of complex
entities (or entities with complex functions), and there may be several hundred of
them. To obtain the required kind of overall evaluative conclusion, it is typically
necessary to combine all of them by means of what is called 'the internal synthesis
process.' This synthesis step is one of the key logical processes in evaluation and
is a long way from the simple deduction and statistical inference that are more
common elements in scientific inference.
Factual premises in a field like program evaluation are commonly established
using the standard procedures of social science, sometimes with the assistance of
other disciplines such as history and jurisprudence. Value premises usually come
from one or more of eight sources: legal principles; scientific and mathematical
standards of truth (especially relevant when the program disseminates information
or is based on scientific theories or common assumptions); professional, cultural,
or organizational standards of proper conduct (e.g., the APA testing standards);
needs assessments; definitions (which provide linguistic standards of propriety);
market research; logic; and ethics. Again, the social sciences are a common source
for several of these types of values e.g., the scientific standards of truth used in
judging the quality of the assertions--or assumptions--built into or propagated by
the program. From psychology, we frequently encounter premises about
maturational rates, cognitive processes, or leadership research. The logic of
evaluation comes in with the frequent need to balance these value considerations
when they conflict: that logic originates in jurisprudence and moral reasoning, but
has been expanded to cover other fields of evaluation besides these, e.g.,
evaluation of alternatives in high-stakes decision-making.
Evaluation Fields
The better-known fields of applied evaluation vary considerably in quality as well
as in their relevance to and dependence on the social sciences. While most of
program and personnel evaluation is heavily dependent on the social sciences and
capable of a high degree of objectivity and utility, others vary independently on
these dimensions. Some come close to being pseudo-evaluative (e.g., wine tasting,
art criticism), some are partially valid (architectural criticism, portfolio
management, literary criticism), and some support highly valid evaluations but are
not dependent on the social sciences (e.g., the reviews done by appellate courts,
the evaluations of claimed proofs of Fermat's Theorem in mathematical journals).
Eight of these applied fields are of particular importance, for practical or logical
reasons: they fall into two groups. The 'Big Six' are the fields of program,
personnel, performance, policy, proposal, and product evaluation (the latter
including technology assessment). The 'Super Two' are:
- intradisciplinary
evaluation--the evaluation of the entities that are the currency of a discipline's
economy such as theories, hypotheses, classifications, data, research designs and
results, practitioners, contributions, and journals--and of the discipline itself: and
- meta-evaluation, the evaluation of evaluations themselves, a practice that
demonstrates the reflexive nature of evaluation and the reflective integrity of its
practitioners.
The first is the backbone of all disciplines--it is what makes them
disciplines. The second is the backbone of evaluation--it is what makes it
consistent by making it practice what it preaches.
Of the Big Six fields--the conventional fields of evaluation--program
evaluation is the one with the largest associated job market at the moment, with
personnel evaluation (an Human Resources staple) and performance evaluation
(especially in the educational area, where its academic fountain is often referred to
as 'tests and measurement') coming next.
Additional Reading
Chelimsky, E and Shadish W.R (eds) (1997) Evaluation for the 21st Century : A
Handbook. Sage Publications. [amazon]
Joint Committee on Standards for Educational Evaluation (1998). Program
Evaluation Standards : How to Assess Evaluations of Educational Programs.
Corwin Press. [amazon]
Scriven, M. (1991) Evaluation Thesaurus 4th edition. Sage Publications.
[amazon]
Shadish W. R. (Chair) (1998) Guiding Principles for Evaluators. A Report from
the Americian Evaluation Association Task Force on Guiding Principles for
Evaluators. [available online http://www.eval.org/EvaluationDocuments/aeaprin6.html].
Shadish, William (1998). Some Evaluation Questions.
Practical Assessment, Research & Evaluation, 6(3). [Available online:
http://ericae.net/pare/getvn.asp?v=6&n=3].
|
