Gail S. MacColl & Kathleen D. White
Program Evaluation and Methodology Division
U.S. Government Accounting Office

Parents, educators, school board members, and legislators all want to know "what works" and "what doesn't" in terms of educational programs and innovations. The reasons for their interest are obvious and worthwhile: first, they want to be sure that tax money is being spent on educational programs that provide a positive return in terms of student progress; second, they want to stay informed of trends in education so they know that their school districts are keeping up with the latest practices and programs.

This digest describes some of the problems in communicating with these audiences; it then provides helpful information on how researchers can best present data on educational practices that work and those that don't to these various audiences for maximum effectiveness, impact, and influence and to keep communication with these audiences open and valuable.

Problems in effective communication to general, nontechnical audiences

Accessibility

Most research on effective educational practices does not filter down to the people who contribute to or control funding. The main reason for this is that research reports on educational practices almost universally appear only in professional and academic journals or through other specialized sources.

The average reader wanting to learn about successful innovations in education is generally unable to locate such information, even after expending considerable effort. These kinds of reports are usually unavailable through popular periodicals or bookstore chains and rarely through more "serious" bookstores; in addition, they are not often found in or through local libraries, including those in large metropolitan areas.

Readability

In the rare event that a general reader gains access to materials about workable educational programs, three stylistic characteristics of these reports often make them unappealing: organization, terminology, and presentation of statistical data.

First, research studies are often organized in such a way as to hide major findings and conclusions in the text or present them only at the end. A related problem is that abstracts and introductions do not provide findings. Even diligent readers become discouraged by these factors because the usefulness of a report or study is not readily apparent.

Second, although the use of technical terminology often simplifies communication within a discipline, it creates an obstacle for policy makers, parents, and other interested readers, who usually are not trained in research or statistical techniques needed to understand an esoteric research study.

Third, many research studies use complex tables to summarize statistical data. These tables, like research jargon, are often difficult for interested parents and program funders to interpret, even with considerable effort.

Difficulties in reporting "what doesn't work"

Researchers often have difficulty in reporting on educational practices that don't work, generally for one or more of the following reasons:

• This question is often addressed only indirectly because most studies reports on something that works, only implying that its absence or the presence of its opposite doesn't work. Unless it is strongly and directly stated, the message that something is ineffective rarely comes out.
• No broad agreement exists about the meaning of "a practice doesn't work." First, "practice" is defined either very broadly or in ideal terms, not in any generally accepted way; second "doesn't work" could have any of several meanings:
• The "practice" is difficult or impossible to implement as intended.
• It has not succeeded in most places where it has been tried.
• It is associated with generally negative results or minimally positive ones.
• It generates fewer positive results than alternative practices.

• Evidence that a practice doesn't work is rarely unequivocal, in part because the results it generates may change in different contexts or with slight modification.

Technical weaknesses that limit usefulness

Assuming that the other problems are overcome, several weaknesses can occur in the research itself to limit its value to those funding, evaluating, or deciding on the use of new educational programs:

• Design constraints--Many reports are based on single case studies. This factor limits a study's applicability and its generalizability.
• Insufficient demographic data and contextual background--When this weakness appears in an article, general readers becomes concerned about the applicability of a practice in their situations.
• Lack of findings in terms of student progress--The few studies that include these data used limited outcome measures, such as norm-referenced tests and SAT scores to measure classroom achievement.
• Little guidance in improving practices in meta-analyses--Although often effective in summarizing a number of studies, meta-analyses in these studies generally omit important details for determining the value of individual studies.
• Policy statements disguised as objective research--Readers must exercise care to separate useful studies from these, which are supported by both shaky assumptions and selective data; unfortunately, with the growing popularity of the Internet, these will probably be found with increasing frequency on the Internet.

How to increase the value of research studies to a wider audience

Most of the recommendations for making research data more useful to more people are simple, relatively easy to accomplish, and based on common sense.

In general, the primary things to focus on are the needs of the audience. Researchers must remember that, in order for their data to be most useful, they have to be accessible and understandable to people with vested interests in the education process: parents, teachers, legislators, school board members. These audiences either pay for, deliver, or fund education programs, and each wants the best ones available.

Researchers uncovering and reporting on programs and practices that work need to distribute their findings as widely, clearly, and efficiently as they can; otherwise their efforts do not create the levels of benefits for the discipline of education that they might.

More specifically, when reporting study results to nontechnical audiences, researchers should keep the following suggestions in mind:

• Summarize the findings in plain language at the beginning of the report. Most nonresearchers appreciate getting the meat of the matter quickly without having first to trim away the fat.
• Present the information in a manner that allows it to be absorbed quickly. As with most of us, even the most interested general readers have time constraints. The more a researcher can do to help readers overcome this problem, the more that he or she will benefit the future of education.
• Provide more detailed material later in a report for those wanting it, but not in place of the summary data.
• Communicate through channels that reach the general public.

To accomplish these goals, researchers will have to learn how to creatively present their findings not only to reach more general readers but to appeal to them too. This requires several steps:

• Simplifying language so that readers without backgrounds in research or statistics can readily understand the content of a report.
• Creating simple tabular material that readers can more easily interpret than dense statistical tables sometimes found in scholarly research journals.
• Incorporating inviting graphics into materials intended for general audiences. These tend to encourage reading and help reader understanding of the material.
• Enlisting the aid of journalists and other communicators who can help both in designing the information for mass consumption and in placing the information in media that the general reader will see.
• Publishing on the Internet, an extraordinarily powerful tool for making information accessible to a wide audience.
• Making certain that the research supports your conclusions, that the work contributes to advancing the level of education, and that a critical eye was used to examine the purpose, the objectivity, and the methodology behind the study.

References

Good, R. G. & Wandersee, J. H. (1991). "No Royal Road: More on Improving the Quality of Published Educational Research." Educational Researcher, 20 (8), 24-25.

Newmann, F. M. & Wehlage, G. G. (1995). Successful School Restructuring: A Report to the Public and Educators by the Center on Organization and Restructuring of Schools. Madison, WI: Center on Organization and Restructuring of Schools.

U. S. General Accounting Office. (1995). Program Evaluation: Improving the Flow of Information to the Congress (GAO/PEMD-95-1). Washington, DC.

U. S. General Accounting Office. (1996). Schools and Workplaces: An Overview of Successful and Unsuccessful Practices (GAO/PEMD-95-28). Washington, DC.

Sitemap 1 - Sitemap 2 - Sitemap 3 - Sitemap 4 - Sitemape 5 - Sitemap 6