ERIC Citations for Methodological Issues & Problems in Survey Research
Instructions for ERIC Documents Access
Search Strategy:
Surveys or Questionnaires or Survey Research [as ERIC Descriptors, with heavily weighted status]
AND
Research Methodology or Evaluation Methods or Educational Research [ERIC Descriptors]
AND
Research Problems or Research Design or Test Items or Validity or Reliability or Sampling or Sample Size or Response Rates (Questionnaires) [ERIC Descriptors]
ED420693 TM028368
Blunders and Missed Opportunities in Survey Research.
Johanson, George; Green, Suzy; Williams, James
1998
8p.; Paper presented at the Annual Meeting of the American
Educational Research Association (San Diego, CA, April 13-17, 1998).
Document Type: EVALUATIVE REPORT (142); CONFERENCE PAPER (150)
Errors in methodology occur regularly in the conduct of surveys for
educational research. This paper discusses some of these errors and
alternatives. In the area of survey design, errors are common in:
(1) missed opportunities in attitude scale planning; (2) blunders in
item construction; (3) missed opportunities in item construction; (4)
blunders in scale revision; and (5) missed opportunities in scale
revision. In the area of survey analysis, there are blunders and
missed opportunities in outlier disposition, as there are in
nonresponse. With regard to the interpretation of survey results,
blunders in causal conclusions and missed opportunities with true
experiments are common. This list is far from complete, but it does
expose some of the more blatant errors in survey research. A
researcher cannot adequately correct earlier errors with later
procedures, and so should be attentive throughout the entire survey
research process. (Contains 17 references.) (SLD)
Descriptors: *Questionnaires; Research Design; *Research
Methodology; Research Problems; *Surveys
Identifiers: Outliers
ED419839 TM028415
Words and Their Value to the Survey Researcher.
McCall, Chester H.; Walters, Lauren E.
1998
27p.; Paper presented at the Annual Meeting of the American
Educational Research Association (San Diego, CA, April 13-17, 1998).
Document Type: RESEARCH REPORT (143); CONFERENCE PAPER (150)
In May 1972, 2,900 copies of a survey were distributed eliciting
opinions about the meaning of some qualifying adjectives and phrases
commonly used in "verbal communications." These same adjective
phrases are commonly used in survey research. They include phrases
such as "a few," "a majority," "nearly all," and other quantifiers.
The 1972 survey had a 40% response rate, indicative of interest in
the study. This study updates that 1972 study, reporting findings
from different groups of subjects. Nine groups (graduate students in
several disciplines, including education, undergraduates, and
engineers and administrators) ranging in size from 3 to 20 members
were surveyed. Results made it clear that individuals within
different groups give different interpretations to quantifying
phrases, and that differences also exist among groups. It is
suggested that there is a need to study item reliability in surveys
in more detail if words and phrases, such as the 22 identified in
this study, are to be included in response options, narrative
instructions in a survey, or reporting on the survey findings.
Appendix A is the report from the 1972 survey, and Appendix B
contains a copy of the original survey form. (Contains five tables,
eight figures, and three references.) (SLD)
Descriptors: Administrators; *Definitions; Engineers; *Graduate
Students; Higher Education; *Research Methodology; Responses;
*Surveys; Tables (Data); *Test Construction; Test Items;
*Undergraduate Students
Identifiers: *Quantifiers
ED416248 TM028128
Improving the Mail Return Rates of SASS Surveys: A Review of the
Literature. Working Papers Series.
Cole, Cornette; Palmer, Randall; Schwanz, Dennis
Bureau of the Census (DOC), Suitland, MD.; National Center for
Education Statistics (ED), Washington, DC. 1997
42p.
Available From: U.S. Department of Education, Office of Educational
Research and Improvement, National Center for Education Statistics,
555 New Jersey Avenue, N.W., Room 400, Washington, DC 20208-5654.
Document Type: REVIEW LITERATURE (070); STATISTICAL MATERIAL (110);
EVALUATIVE REPORT (142)
This paper presents results of a review of mail survey response
literature and provides recommendations for the application of some
response-enhancing techniques to the Schools and Staffing Survey
(SASS) of the National Center for Education Statistics. The SASS is
an integrated set of surveys sponsored by the NCES and conducted by
the Bureau of the Census to provide data on teachers, students,
administrators, librarians, and libraries in public and private
schools. The SASS uses mail questionnaires as its primary method of
data collection, but in the most recent survey, only between 25% and
75% of surveys have been returned. It would not have been wise to
produce SASS estimates based on the responses from mail surveys alone.
After telephone interviews and a few face-to-face interviews, the
response rate rose to between 80% and 100%, but obviously costs would
be much reduced with better mail responses. A review of the
literature indicates that response rate enhancement techniques have
been grouped into the following general categories: (1) motivating a
response; (2) content and appearance of correspondence; (3) postage
supplied; and (4) attitudinal and behavioral modifications (not
discussed in this report). Research indicates that multiple contacts
are the most effective way to increase response rates, and that the
SASS already includes most multiple contact techniques, with the
exception of the use of special postage. Use of a fifth mail contact
with two-day priority delivery is recommended. Another successful
technique has been the inclusion of monetary incentives, a technique
that does not seem appropriate for the SASS. The use of nonmonetary
incentives, such as thank-you cards, is recommended. Making the
questionnaires briefer and easier to complete is among additional
recommendations. (Contains 7 tables and 24 references.) (SLD)
Descriptors: Costs; Data Collection; Elementary Secondary Education;
Incentives; Literature Reviews; *Mail Surveys; *Questionnaires;
Research Design; *Research Methodology; *Response Rates (
Questionnaires); Tables (Data)
Identifiers: *Schools and Staffing Survey (NCES)
EJ533536 TM519738
Advances in Survey Research.
Braverman, Marc T., Ed.; Slater, Jana Kay, Ed.
New Directions for Evaluation, n70 p1-104 Sum 1996
Document Type: SERIAL (022); EVALUATIVE REPORT (142); JOURNAL
ARTICLE (080)
The seven articles of this special issue focus on theory and
research related to survey methods and whether evaluators need
information that is absent from the larger survey literature. These
articles reflect the view that the considerations relating to good
survey research apply also to using surveys in evaluation. (SLD)
Descriptors: Data Collection; *Evaluation Methods; *Evaluators;
Needs Assessment; *Program Evaluation; Public Opinion; Questionnaires;
Research Design; *Research Methodology; Research Utilization;
Response Rates (Questionnaires); *Surveys; Test Construction;
Theories
Identifiers: Rasch Model
ED399293 TM025607
Differential Item Functioning in Survey Research.
Johanson, George A.; Johanson, Susan N.
Apr 1996
9p.; Paper presented at the Annual Meeting of the American
Educational Research Association (New York, NY, April 8-12, 1996).
Document Type: EVALUATIVE REPORT (142); CONFERENCE PAPER (150)
Differential item functioning (DIF), or item bias, occurs when
individuals in a focal group respond differently to a test item than
do individuals in a reference group even when comparisons are
restricted to individuals with similar overall skill levels on the
trait in question. It is common in constructing a questionnaire or
survey to recommend that an item analysis be conducted in a manner
similar to that used in cognitive measurement, but it is not common
to be concerned with items as they perform differently. DIF has
apparently not yet been widely recognized as a tool for developing a
survey or for understanding survey responses. The Mantel Haenszel
procedure is one of the empirical methods most commonly used to
identify DIF, and its use in survey development is explored. Two
examples, one involving the evaluation of student achievement for 777
male and 773 female elementary school students, and the other
involving the evaluation of human service workshops for 798
participants aged over 40 years and 884 who were younger, illustrate
the way in which information about DIF could have aided in the
development of the instrument and interpretation of the data. DIF
detection would seem a useful adjunct to the traditional item
analysis that could be of substantial value at the pilot or revision
stage of instrument development. (Contains 4 figures and 20
references.) (SLD)
Descriptors: Achievement Tests; Data Analysis; *Evaluation Methods;
Item Analysis; *Item Bias; *Research Methodology; *Surveys; *Test
Construction; Test Items
Identifiers: Item Bias Detection; *Mantel Haenszel Procedure
ED397740 HE029340
Balancing Resources and Response Rates in Mailed Questionnaire
Research: The Challenge of Conducting Successful Alumni Surveys While
Containing Costs. AIR 1996 Annual Forum Paper.
Ahson, Nancy L.; Gentemann, Karen M.
May 1996
37p.; Paper presented at the Annual Forum of the Association for
Institutional Research (36th, Albuquerque, New Mexico, May 5-8,
1996).
Document Type: CONFERENCE PAPER (150); NON-CLASSROOM MATERIAL
(055); TEST, QUESTIONNAIRE (160)
This paper describes empirically tested strategies for containing
mailed questionnaire costs while maximizing survey response rates.
Survey focus, critical to gaining respondent interest and completion
of the survey instrument, should be selected on the basis of the
theme of the study in order to reduce survey length and yield more in-
depth and potentially useful information on selected topics. To help
insure that the survey focus interests respondents, the appropriate
study population should be carefully selected. Psychometric
considerations should determine whether addressing a random sample or
entire population is cost effective. Appearance and organization of
the survey, as well as its form, style and presentation should be
tailored to arouse the interest of the target group. The mailing
process should include a premailing letter or phone call, an initial
mailing, a postcard mailing, a second cover letter and replacement
questionnaire, and a certified mailing or telephone follow-up. Two
case studies revealing differential successes with variations to
traditional survey research are reported and sample surveys are
appended. (CK)
Descriptors: Case Studies; *Cost Effectiveness; Data Collection;
*Graduate Surveys; Higher Education; *Institutional Research; *Mail
Surveys; Psychometrics; *Questionnaires; Research Methodology;
Response Rates (Questionnaires); Sample Size; Statistical Analysis;
Telephone Surveys
Identifiers: *AIR Forum; Cost Containment; *Survey Research
ED387018 HE028599
Working with Low Survey Response Rates: The Efficacy of Weighting
Adjustments. AIR 1995 Annual Forum Paper.
Dey, Eric L.
Michigan Univ., Ann Arbor. Center for the Study of Higher and
Postsecondary Education. May 1995
20p.; Paper presented at the Annual Forum of the Association for
Institutional Research (35th, Boston, MA, May 28-31, 1995).
Document Type: EVALUATIVE REPORT (142); CONFERENCE PAPER (150)
Problems associated with low response rates to surveys are
considered, drawing from the literature on the methodology of survey
research. A series of analyses are presented which were designed to
examine the efficacy of Astin and Molm's procedure to adjust for
nonresponse biases. Data were obtained form the Cooperative
Institutional Research Program survey of 1987 incoming students and
the 1991 followup survey. Data were analyzed for 209,627 students at
390 institutions. After separating the data into respondents (21
percent) and nonrespondents (79 percent), attention was directed to
how distributions and relationships change due to nonresponse. The
efficacy of the Astin and Molm procedure was examined using
univariate and joint distributions and multiple regressions. It is
concluded that the Astin and Molm procedure for adjusting for
patterns of nonresponse is very effective in reducing biases in
univariate distributions. The effectiveness of this weighting
procedure is less clear in adjusting the results of correlation and
regression analyses. The analyses did not reveal a situation where
the weighted results were clearly less preferable than the unweighted
one. Although the Astin and Molm technique is designed for use in
longitudinal research, related methods of weighting can be
implemented for cross-sectional surveys. (Contains 27 references.)
(SW)
Descriptors: *Followup Studies; Higher Education; *Institutional
Research; *Longitudinal Studies; *Response Rates (Questionnaires);
Statistical Bias; *Surveys
Identifiers: *AIR Forum; Astin and Molm Procedure; Cooperative
Institutional Research Program; *Survey Research
ED387011 HE028592
Evaluating Reasons for Low Response from Mail Surveys. AIR 1995
Annual Forum Paper.
Westcott, S. Wickes, III; And Others
May 1995
18p.; Paper presented at the Annual Forum of the Association for
Institutional Research (35th, Boston, MA, May 28-31, 1995).
Document Type: RESEARCH REPORT (143); CONFERENCE PAPER (150);
TEST, QUESTIONNAIRE (160)
A study was undertaken to solicit opinions from alumni on methods
that might improve responses from graduate surveys. Two telephone
surveys were conducted, one in 1991 which targeted the graduating
classes of 1984 and 1989, and the second in 1994 among alumni of the
classes of 1991 and 1993. In the 1994 survey information was
gathered regarding factors that affected forwarding of mail surveys.
Study findings indicated that low response may have been related to
out-of-date addresses, which resulted in surveys not being correctly
delivered. Opinions were obtained on other aspects of the alumni
survey which could be adjusted to improve the response rate. Fifty-
one percent of the 1994 respondents found the number of pages of the
survey influential in whether they returned the survey, while only 45
percent of the 1991 sample felt this way. The importance of the
questions to the respondent and the fact that the university was
interested in their input were frequently indicated as factors in
their decisions to respond. About half of the alumni contacted in
both surveys felt that an incentive would influence them to return
the survey. Three fourths of the 1991 respondents felt that their
opinions would make a difference, while only just over one half of
the 1994 respondents felt that way. Two survey interviews are
appended. (Contains 22 references.) (SW)
Descriptors: *Alumni; *Graduate Surveys; Higher Education;
*Institutional Research; *Response Rates (Questionnaires)
Identifiers: *AIR Forum; *Survey Research
ED391843 TM024659
Reexamining Traditional Issues in Survey Research: Just How Evil Is
the Anathema of Low Response Rate?
Clark, Sheldon B.; Boser, Judith A.
Apr 1995
15p.; Paper presented at the Annual Meeting of the American
Educational Research Association (San Francisco, CA, April 18-22,
1995).
Document Type: RESEARCH REPORT (143); CONFERENCE PAPER (150)
This study was designed to determine the extent to which the
results of an employment survey of graduates of a teacher preparation
program would have been affected by changes in response rate. At the
University of Tennessee, Knoxville, a followup of teacher education
program graduates is conducted annually. A total of 284 graduates of
the 1992 teacher education program were identified as the target
population for the 1993 survey. A total of 184 individuals responded
to the mail survey (64.8%), with 33.8% responding to the first wave,
17.2% responding to the second, 6.3% to the third wave, and 7.4%
responding late. Telephone calls elicited information from 40
additional persons (14.1%). Supplemental information about
employment from other sources resulted in the eventual determination
of the occupations of 265 of the 284 individuals (93.3%). There was
no evidence that data collected after about 50% of the sample had
responded resulted in any meaningful differences in survey results.
These results suggest that concentrating on potential nonresponse
bias may not be as important as attending to other aspects of survey
methodology, such as sample size and questionnaire design. (Contains
2 tables and 28 references.) (SLD)
Descriptors: Data Collection; Employment Patterns; *Graduates;
Higher Education; *Research Methodology; *Response Rates (
Questionnaires); Responses; Sample Size; Statistical Bias; *Surveys;
*Teacher Education; *Vocational Followup
Identifiers: Missing Data; *Nonresponse Bias; University of
Tennessee Knoxville
ED357069 TM019848
A Critical Analysis of Interview, Telephone, and Mail Survey
Designs.
Katz, Elinor
Apr 1993
19p.; Paper presented at the Annual Meeting of the American
Educational Research Association (Atlanta, GA, April 12-16, 1993).
Document Type: REVIEW LITERATURE (070); EVALUATIVE REPORT (142);
CONFERENCE PAPER (150)
A critical analysis is presented of the literature as it relates to
survey research, including personal interviews, telephone interviews,
and mail questionnaires. Additional research concerns are explored,
and a code of ethics for survey researchers is presented. Focus
groups, interviews, long interviews, telephone interviews, and mail
surveys are considered; and the advantages and drawbacks of each are
noted. Other research concerns which have an impact on survey
research include sources of error, the nature of bias, and response
rates. The code of ethics for survey research emphasizes that
subjects should volunteer their participation, and the researcher
should assure that no harm comes to them as a result of participation.
Participants, who should be assured of anonymity and confidentiality
if these are guaranteed, should know the purpose of the study and who
the sponsors are. Respondents should be able to contact the
researcher. Analysis and reporting should provide an honest
accounting of the procedures, and unexpected findings should be
acknowledged. "A Guide for Survey Research," a summary of advantages
and disadvantages, is included. A chart summarizes comparisons of
mail, face-to-face, and telephone surveys. (SLD)
Descriptors: *Codes of Ethics; Comparative Testing; Confidentiality;
*Interviews; Literature Reviews; *Mail Surveys; *Research Design;
*Researchers; Research Methodology; Response Rates (Questionnaires);
Responses; *Telephone Surveys; Test Bias
Identifiers: *Focus Groups Approach; Researcher Subject
Relationship; Survey Research
EJ488661 SO525610
Data Management in Educational Survey Research.
Schleicher, Andreas; Umar, Jahja
Prospects, v22 n3 p317-25 1992
ISSN: 0033-1538
Available From: UMI
Document Type: PROJECT DESCRIPTION (141); JOURNAL ARTICLE (080)
Target Audience: Researchers; Policymakers
Asserts that educational policymakers must measure student
achievement and school system performance accurately to help provide
skills demanded by society. Contends that many international surveys
either fail to provide adequate and precise answers to the intended
research questions or provide inaccurate and faulty results. (CFR)
Descriptors: *Academic Achievement; *Comparative Education; Cross
Cultural Studies; Data Interpretation; *Educational Research;
Elementary Secondary Education; Foreign Countries; Government School
Relationship; International Educational Exchange; *Research Design;
Research Methodology; *Research Problems
Identifiers: *International Assn Evaluation Educ Achievement;
Survey Research
EJ488658 SO525607
Managing International Survey Research.
Loxley, William
Prospects, v22 n3 p289-96 1992
ISSN: 0033-1538
Available From: UMI
Document Type: PROJECT DESCRIPTION (141); JOURNAL ARTICLE (080)
Target Audience: Researchers; Policymakers
Discusses administrative policies related to projects of the
International Association for the Evaluation of Educational
Achievement (IEA). Describes a typical budget for an IEA project.
Includes a figure illustrating a typical project timetable. (CFR)
Descriptors: *Academic Achievement; *Comparative Education; Cross
Cultural Studies; *Educational Administration; *Educational
Innovation; *Educational Research; Educational Researchers;
Elementary Secondary Education; Financial Support; Higher Education;
International Educational Exchange; International Organizations;
Research and Development
Identifiers: *International Assn Evaluation Educ Achievement;
Survey Research
ED325492 TM015709
The Robustness of Regression and Substitution by Mean Methods in
Handling Missing Values.
Kaiser, Javaid
Aug 1990
24p.; Paper presented at the Annual Islamic Conference on
Statistical Sciences (2nd, Johor Bahru, Malaysia, August 26-30,
1990).
Document Type: CONFERENCE PAPER (150)
There are times in survey research when missing values need to be
estimated. The robustness of four variations of regression and
substitution by mean methods was examined using a 3x3x4 factorial
design. The regression variations included in the study were: (1)
regression using a single best predictor; (2) two best predictors;
(3) all available preditors having observed values; and (4) all
available predictors with adjustment of estimate for predictors
having missing values. The factors studied included sample size
(n=30, 60, and 120), the proportion of incomplete records (IRs) in
the sample (IR=10%, 20%, and 30%), and the number of missing values
(MVs) per IR (MVs=12.5%, 25%, 37.5%, and 50%). The design matrix was
replicated 500 times. Imputation methods were compared in terms of
retaining population covariance structure in imputed samples. The
results suggest that all methods significantly altered covariance
structure and that the regression variation that adjusts missing
value estimates for predictors having missing values was found to be
the best imputation method at all experimental conditions. Three
data tables and six graphs are included. (TJH)
Descriptors: Comparative Analysis; Computer Simulation; Estimation
(Mathematics); *Predictor Variables; *Regression (Statistics);
Research Problems; *Robustness (Statistics); Sample Size; Surveys
Identifiers: Covariance Structural Analysis; *Missing Data; Survey
Research
ED317606 TM014715
The Noggin Factor in Survey Research: Developing New Techniques for
Assessing Nonresponse Bias.
Clark, Sheldon B.
Oak Ridge Associated Universities, Tenn. Apr 1990
12p.; Paper presented at the Annual Meeting of the American
Educational Research Association (Boston, MA, April 16-20, 1990).
Sponsoring Agency: Department of Energy, Oak Ridge, TN. Technical
Information Center.; National Science Foundation, Washington, D.C.
Document Type: RESEARCH REPORT (143); CONFERENCE PAPER (150)
Target Audience: Researchers
The primary objective of this paper is to encourage survey
researchers not to become overly reliant on the literature for
generic solutions to non-response bias problems. In addition, the
paper recounts an example of how a non-traditional approach was used
to maximize the usefulness of data collected under unusual
constraints and with an a priori expectation of a high rate of non-
response. The author was charged with testing the ability of the
National Science Foundation to conduct a biennial survey of
scientists and engineers who had recently immigrated into the United
States. This assessment was done within the context of the "noggin
factor," which engenders a consideration of the particularity or each
survey situation. As a result of the noggin factor, generalized
solutions to survey research problems tend to be self-defeating. It
is concluded that survey researchers should attend more to process
than to outcome. (TJH)
Descriptors: Engineers; Immigrants; National Surveys; *Research
Problems; Scientists; Statistical Bias; *Statistical Surveys
Identifiers: National Science Foundation; *Noggin Factor;
*Nonrespondents; Survey Research
ED308224 TM013590
Monetary Gratuities in Survey Research: A Meta-Analysis of Their
Effects on Response Rates.
Hopkins, Kenneth D.; Gullickson, Arlen R.
Mar 1989
10p.; Paper presented at the Annual Meeting of the American
Educational Research Association (San Francisco, CA, March 27-31,
1989).
Document Type: CONFERENCE PAPER (150); RESEARCH REPORT (143)
The effects of monetary gratuities on response rates to mail
surveys have been considered in a number of studies. This meta-
analysis examined: (1) the nature of the population surveyed; (2) the
effects of gratuities in relation to the number of follow-ups; (3)
whether the gratuity was equally effective across different
populations; (4) whether the gratuity was promised or enclosed; and
(5) the year of publication of the study. The bulk of the studies
was done in the context of market research. Findings indicate that a
monetary gratuity can be a powerful tool for decreasing non-response
bias in mail surveys. Although follow-up mailings generally
increased response rates in themselves, the gratuity typically had
more impact than did an additional mailing. The gratuity was almost
equally effective for professional and general populations.
Enclosing the gratuity was more effective than was promising it upon
return of the questionnaire. The year of publication had some effect
in that the impact of the gratuity was slightly greater in earlier
studies. The additional expense of the gratuity to improve response
rates appeared cost-effective for most purposes. These findings have
clear implications for survey practice. When differences in response
with and without gratuities were found, they tended to show greater
response integrity in the gratuity-receiving group. (SLD)
Descriptors: Comparative Analysis; Evaluation Methods; *Mail
Surveys; Meta Analysis; *Questionnaires; *Research Methodology;
Responses; Statistical Bias; Testing Problems
Identifiers: Followup Materials; Gratuities; Market Research;
*Monetary Incentives; Nonresponders; *Response Rates (Questionnaires);
Survey Research
ED308208 TM013423
Interpreting Nonresponse in Survey Research: Methodological Heresy?
Clark, Sheldon B.; Finn, Michael G.
Oak Ridge Associated Universities, Tenn. Mar 1989
17p.; Paper presented at the Annual Meeting of the American
Educational Research Association (San Francisco, CA, March 27-31,
1989).
Sponsoring Agency: Department of Energy, Washington, D.C.; National
Science Foundation, Washington, D.C.
Document Type: CONFERENCE PAPER (150); RESEARCH REPORT (143)
A study is proposed that seeks to use the normally problematic
factor of non-response to a survey in a positive way in order to
estimate certain characteristics of a population subgroup. A
longitudinal database, the Scientific and Technical Personnel Data
System (STPDS) measures the educational, demographic, and employment
characteristics of the nation's scientists and engineers. An
examination of response trends for STPDS surveys reveals that
response rates for foreign-born scientists and engineers deteriorate
at a faster rate over time than do the response rates of those born
in the United States. It is hypothesized that emigration accounts
for these differences, and that the difference in the rates of
decline of response rates can be used as a proxy for emigration of
scientists. Two very different approaches have been developed and
will be used to test the hypothesis and to validate the proposed
technique. Although this methodology is not generalizable to all
surveys, it does illustrate the importance of a researcher's being
thoroughly familiar with the survey population and the importance of
being open to the possibility that challenging and testing the
precepts of the practice of survey research can result in
improvements in methodology. Three bar graphs give information about
the proposed study. (SLD)
Descriptors: Engineers; Evaluation Methods; Occupational Surveys;
*Research Methodology; Research Projects; *Scientists; *Surveys; Test
Interpretation
Identifiers: Emigration; *Nonresponders; Response Patterns;
Response Rates (Questionnaires); *Survey Research
ED327568 TM016000
Limiting Nonresponse in Longitudinal Research: Three Strategies for
School-Based Studies. A RAND Note.
Ellickson, Phyllis L.
Rand Corp., Santa Monica, Calif. Mar 1989
19p.
Sponsoring Agency: Conrad N. Hilton Foundation, Los Angeles, CA.
Available From: RAND Corporation, 1700 Main Street, P.O. Box 2138,
Santa Monica, CA 90406-2138.
Document Type: RESEARCH REPORT (143)
Target Audience: Practitioners; Researchers
Longitudinal studies conducted with children typically experience
non-response problems that can make substantial inroads on sample
size and introduce bias into the analysis. Three methods for
limiting non-response in school-based research programs were
assessed: (1) asking parents to return consent forms only if they do
not want their children to participate in the research; (2)
conducting make-up sessions for students who are absent during the
scheduled data collection day; and (3) tracking students who transfer
out of the participating schools. These methods were used in a large-
scale intervention study, involving nearly 10,000 junior high school
students, designed to assess the effects of a smoking and drug use
prevention curriculum for adolescents (Project ALERT) across 30
schools in 8 school districts in California and Oregon. The
evaluation data, which consist of surveys and saliva samples, were
collected in the classrooms before the 7th grade curriculum was
delivered and 3, 12, 24, and 36 months after it was delivered. Each
method substantially reduced the anticipated loss of participating
students over a 2-year period; each method also reduced attrition
bias attributable to the loss of at-risk students (those with
characteristics such as poor grades, prior deviance, and past drug
experience that made them likely candidates for using drugs in the
future). By using a multistage parent consent process that included
multiple communication channels, the risk of mistakenly treating non-
response as consent was minimized, and the loss of between 40% and
50% of the targeted population at the onset was avoided. Over
several waves of data collection, make-up sessions reduced the losses
attributable to absenteeism by an average of about 60%, recapturing
close to 6% of the baseline sample each year. The tracking process
retrieved surveys from two-thirds of the transferees, and cut the
overall attrition rate in half. Two tables and one figure illustrate
the study. (SLD)
Descriptors: Adolescents; *Attrition (Research Studies); Curriculum
Evaluation; Data Collection; Drug Education; Health Education; High
Risk Students; Junior High Schools; *Junior High School Students;
*Longitudinal Studies; *Research Methodology; Research Problems;
*Response Rates (Questionnaires); Sampling; Smoking; Statistical Bias
Identifiers: *Nonresponders; Project ALERT; *Student Surveys
ED308208 TM013423
Interpreting Nonresponse in Survey Research: Methodological Heresy?
Clark, Sheldon B.; Finn, Michael G.
Oak Ridge Associated Universities, Tenn. Mar 1989
17p.; Paper presented at the Annual Meeting of the American
Educational Research Association (San Francisco, CA, March 27-31,
1989).
Sponsoring Agency: Department of Energy, Washington, D.C.; National
Science Foundation, Washington, D.C.
Document Type: CONFERENCE PAPER (150); RESEARCH REPORT (143)
A study is proposed that seeks to use the normally problematic
factor of non-response to a survey in a positive way in order to
estimate certain characteristics of a population subgroup. A
longitudinal database, the Scientific and Technical Personnel Data
System (STPDS) measures the educational, demographic, and employment
characteristics of the nation's scientists and engineers. An
examination of response trends for STPDS surveys reveals that
response rates for foreign-born scientists and engineers deteriorate
at a faster rate over time than do the response rates of those born
in the United States. It is hypothesized that emigration accounts
for these differences, and that the difference in the rates of
decline of response rates can be used as a proxy for emigration of
scientists. Two very different approaches have been developed and
will be used to test the hypothesis and to validate the proposed
technique. Although this methodology is not generalizable to all
surveys, it does illustrate the importance of a researcher's being
thoroughly familiar with the survey population and the importance of
being open to the possibility that challenging and testing the
precepts of the practice of survey research can result in
improvements in methodology. Three bar graphs give information about
the proposed study. (SLD)
Descriptors: Engineers; Evaluation Methods; Occupational Surveys;
*Research Methodology; Research Projects; *Scientists; *Surveys; Test
Interpretation
Identifiers: Emigration; *Nonresponders; Response Patterns;
Response Rates (Questionnaires); *Survey Research
EJ346459 TM511607
Sample Design.
Ross, Kenneth N.
International Journal of Educational Research, v11 n1 p57-75
1987
Theme issue with title "Educational Evaluation: The State of the
Field."
Document Type: JOURNAL ARTICLE (080); PROJECT DESCRIPTION (141)
This article considers various kinds of probability and non-
probability samples in both experimental and survey studies.
Throughout, how a sample is chosen is stressed. Size alone is not
the determining consideration in sample selection. Good samples do
not occur by accident; they are the result of a careful design.
(Author/JAZ)
Descriptors: *Educational Assessment; Elementary Secondary
Education; *Evaluation Methods; Experimental Groups; Foreign
Countries; Probability; Research Design; *Sample Size; *Sampling;
*Surveys; Validity
Identifiers: Australia
EJ319353 TM510627
A Note on the Potential for Bias in the Structure and Wording of
Questionnaire Items.
Cantwell, Zita M.
Evaluation News, v6 n2 p9-14 May 1985
Document Type: JOURNAL ARTICLE (080); PROJECT DESCRIPTION (141)
The wording and structure of questionnaire items can interact with
specified sample categories based on evaluation goals and respondent
characteristics. The effects of the interactions can restructure
samples and introduce bias into the data analysis. These effects,
and suggestions for avoiding them, are demonstrated for five types of
questionnaire items. (BS)
Descriptors: Higher Education; *Item Analysis; *Questionnaires;
Statistical Bias; Student Attitudes; *Test Construction; *Testing
Problems; *Test Items
EJ302120 TM508932
The Stages of Mailed Questionnaire Returning Behavior.
Lockhart, Daniel C.
New Directions for Program Evaluation, n21 p89-98 Mar
1984
Theme issue with title "Making Effective Use of Mailed
Questionnaires."
Document Type: POSITION PAPER (120); REVIEW LITERATURE (070)
Six stages are hypothesized that define the behavior of returning
mailed questionnaires: receiving the questionnaire, opening the mail,
forming an overall impression, answering the questions, returning the
questionnaire, and dealing with nonrespondents. The researcher must
provide incentives at each stage if potential respondents are to
complete a mailed questionnaire. (BW)
Descriptors: *Behavior Patterns; *Questionnaires; *Research
Methodology; Surveys; Theories
Identifiers: *Mail Surveys; *Response Rates (Questionnaires)
EJ302118 TM508930
Recent Research on Mailed Questionnaire Response Rates.
Baumgartner, Robert M.; Heberlein, Thomas A.
New Directions for Program Evaluation, n21 p65-76 Mar
1984
Theme issue with title "Making Effective Use of Mailed
Questionnaires."
Document Type: REVIEW LITERATURE (070)
Forty studies of mailed surveys are reviewed in terms of 11
variables which affect response rates: sponsorship, respondents,
salience, follow-up contacts, incentives, length, anonymity,
personalization, deadline, types of appeals, and postage. (BW)
Descriptors: Predictor Variables; *Questionnaires; Regression
(Statistics); *Research Methodology
Identifiers: *Mail Surveys; *Response Rates (Questionnaires)
ED242789 TM840210
Tactics and Factors That Increase Response Rates to Mailed
Questionnaires.
Dillihunt, Vivian C.
22 Mar 1984
15p.
Document Type: REVIEW LITERATURE (070)
Target Audience: Researchers
Research findings on identification of factors associated with
increasing response rates to mailed questionnaires are presented.
Several tactics which have been used to effect a greater response
rate are presented and explained. Physical factors, such as typed
correspondence, have been shown to yield higher response rates than
duplicated correspondence. Pre-questionnaire contact, such as a
telephone call to inform subjects that they will be receiving a
questionnaire, is effective. Use of a personalized cover letter has
proven to be an essential and effective response-soliciting method.
Monetary inducements initiate increases in response rates as do
mailing procedures, such as using stamps on reply envelopes. Follow-
up procedures are effective in increasing response rate. A
compilation of additional suggestions offered by the authors cited in
the literature review is included. (DWH)
Descriptors: *Questionnaires; *Research Methodology; Research
Problems
Identifiers: *Mail Questionnaires; *Response Rates (Questionnaires)
EJ302114 TM508926
Improving Mailed Questionnaires: Analysis of a 96 Percent Return
Rate.
Altschuld, James W.; Lower, Michael A.
New Directions for Program Evaluation, n21 p5-18 Mar
1984
Theme issue with title "Making Effective Use of Mailed
Questionnaires." The survey study described in this chapter was
supported by the Spencer Foundation.
Document Type: NON-CLASSROOM MATERIAL (055); EVALUATIVE REPORT(142)
The procedures used in a successful mailed evaluation questionnaire
effort are described, and guidance is given to those who are
unfamiliar with the methodology. (Author)
Descriptors: Data Collection; Elementary School Teachers;
Elementary Secondary Education; Principals; Questionnaires; *Research
Methodology; Research Problems; School Districts; Secondary School
Teachers; State Surveys; *Surveys
Identifiers: *Mail Surveys; *Response Rates (Questionnaires)
ED279726 TM870175
Validity and Reliability in Survey Research. Technical Report No.
15.
Russ-Eft, Darlene F.
American Institutes for Research in the Behavioral Sciences. Palo
Alto, CA. Statistical Analysis Group in Education. Aug 1980
143p.
Sponsoring Agency: National Center for Education Statistics (ED),
Washington, DC.
Document Type: PROJECT DESCRIPTION (141); BIBLIOGRAPHY (131)
With increasing reliance being placed on the results of their
surveys, the National Center for Education Statistics (NCES)
recognized a need for these survey results to be valid and reliable.
As part of the work of the Statistical Analysis Group in Education
(SAGE), an effort was undertaken to investigate validity and
reliability in survey research. This document is the result of that
effort. The first section provides an overview of the concepts of
reliability and validity. Procedures for measuring sources of error
are suggested. Several ways of approaching validity are mentioned
including content validity, criterion-related validity, and construct
validity. The second section presents some suggestions for approving
the reliability and validity of survey data, with a focus on the data
collection phase. These suggestions are based upon previous data
collection experiences of the staff of SAGE. The final section,
encompassing 106 pages, provides an annotated bibliography of
selected materials relevant to validity and reliability in survey
research. (JAZ)
Descriptors: Concurrent Validity; Construct Validity; Content
Validity; Data Collection; Educational Research; Questionnaires;
*Research Design; *Research Methodology; Research Problems; *Surveys;
*Test Reliability; *Test Validity
Identifiers: Internal Consistency; *Test Retest Reliability
Return to the Index of FAQs
|
