Conducting web-based surveys. Solomon, David J.

David J. Solomon
Office of Medical Education Research and Development and
the Department of Medicine, College of Human Medicine,
Michigan State University

Web-based surveying is becoming widely used in social science and educational research. The Web offers significant advantages over more traditional survey techniques however there are still serious methodological challenges with using this approach. Currently coverage bias or the fact significant numbers of people do not have access, or choose not to use the Internet is of most concern to researchers. Survey researchers also have much to learn concerning the most effective ways to conduct surveys over the Internet. While in its early stages, research on Internet-based survey methodology has identified a number of factors that influence data quality. Of note, several studies have found Internet surveys have significantly lower response rates than comparable mailed surveys. Several factors have been found to increase response rates including personalized email cover letters, follow-up reminders, pre-notification of the intent to survey and simpler formats. A variety of software tools are now available for conducting Internet surveys and they are becoming a increasing sophisticated and easy to use. While there is a need for caution, the use of Web-based surveying is clearly going to grow.

The growth of the Internet has impacted on virtually every aspect of society. Survey research is no exception. Two years ago in an informal search of Yahoo, Kay and Johnson (1999) identified over 2,000 Web-based surveys¹ in 59 areas. The interest in Web-based surveying is not surprising as it offers a number of distinct advantages over more traditional mail and phone techniques. Examples include reducing the time and cost of conducting a survey and avoiding the often error prone and tedious task of data entry (Medin, Roy & Ann, 1999).

Email offers one option for distributing Internet surveys. Up until a few years ago email surveys were the predominate means of Internet surveying. As the World Wide Web (WWW) has grown in popularity, the use of Hypertext Markup Language (HTML) forms or Web-based surveys are becoming the dominant method of gathering survey data. These forms streamline the data collection process formatting and entering responses directly into a database for analysis. Since HTML forms can be made programmable, it is also possible to have real time error checking and correction increasing the accuracy of the data collection process. The formatting capabilities of HTML allow the creation of easy-to-read and attractive forms that may improve response rates. In addition, the programmability of HTML forms makes it possible to randomly order responses and tailor options based on information the respondent supplies earlier in the survey.

Combining an email "cover letter" as a means of contacting sampled people with the use of an HTML form for data collection provides an especially effective and efficient approach to Internet surveying. Modern email packages automatically convert universal resource locators (URLs) or web-addresses in the text of an email into a hyperlinks. Placing the URL of the survey form in a cover letter email allows the respondent to "click" their mouse on the URL to display the survey form and subsequently fill it out.

Although Web-based surveying is very attractive, at this point it should be used with caution. Currently the biggest concern in Internet surveying is coverage bias or bias due to sampled people not having or choosing not to access the Internet (Kay & Johnson, 1999; Crawford, Couper & Lamias, 2001). Despite expediential growth of the Internet there are still large numbers of people who do not have access and/or choose not to use the Internet. It is also clear that there are wide disparities in Internet access among ethnic and socioeconomic groups (Selwyn & Robson, 1998).

There are specific populations where Internet access is extremely high and coverage bias is likely to be less of a concern. College students and university faculty within the USA, Canada and Western Europe are examples of such populations. Even though coverage bias may be less of an issue in these groups, experience and comfort with Internet-based tools such as Web browsers is another serious potential source of bias both in response rates and the way people respond to the survey (Dillman, Tortora & Bowker, 2001).

Web-based surveying is still in the early stages of development. The WWW is a unique media and it is not clear to what extent the knowledge we have gained over years of experience with more traditional surveying techniques fully applies to Internet surveying (Dilman, Tortora & Bowker, 2001). Studies are just beginning to done to learn the optimal ways to structure and format Internet surveys to limit biases and increase response rates. It is also likely that the best way to design an Internet survey depends in part on the familiarity and comfort of the respondents in using Web browsers and email clients. It is also quite likely that the type of Internet connection as well as the hardware and software used in accessing the Internet will impact on response rates and possibly how a person responds to an Internet-based survey.

The use of HTML forms for surveying poses a unique set of issues and challenges that need to be addressed to ensure valid data. The Web is a very public place and unless steps are taken to limit access to a survey, it may be found and responded to by people who are not among those sampled by the researcher. This can either happen by accident or maliciously. Since one only has to "click" their mouse pointer on the "submit" button to respond to a Web-based survey instrument once it is filled out, it is also quite possible for respondents to either mistakenly or purposefully submit multiple copies of their responses.

While Internet-based surveying techniques need to be used with caution, their benefits warrant continued exploration and the cautious use. It is also pretty clear that coverage bias and familiarity with Internet tools will be less and less of an issue over time. Additionally our knowledge about how best to conduct Internet surveys will continue to improve with research and experience.

Although the research on Internet-based surveying is limited, findings are beginning to appear in the literature. Several studies have found that response rates for Internet surveys are lower that equivalent mail surveys (Medin, Roy & Ann, 1999; Cooper, Blair & Triplett, 1999). As noted by Crawford and colleagues (2001), this may be due to our lack of knowledge on how to achieve high response rates using the Internet surveys. The lower response rates for internet surveys may also reflect coverage bias, the lack of familiarity with the media and/or lack of convenient access to the Internet. In the author’s experience, Web congestion can also be a factor in lowering response rates for Web surveys particularly with people who have relatively little experience with the Internet.

Cook and colleagues (2000) conducted a meta-analysis of factors influencing response rates in Internet-based surveys². They found that follow-up contacts with non-respondents, personalized contacts, and contacting sampled people prior to sending out the survey were the three dominant factors in higher response rates. Kittleson (1997) in a study of email-based surveying found it was possible to double the response rate with follow-up memos though in general this may be somewhat optimistic. As with mailed surveys, repeated follow-ups have diminishing returns and at some point risk irritating potential respondents without noticeably increasing response rates. Additionally, Dillman, Tortora, Conrad & Bowker (2001) found that relatively plain Web surveys that load quickly resulted in higher response rates than "fancier" surveys that take longer to load. Jeavons (1998) analyzed detailed server logs from three separate large-scale surveys. He found a relatively high percentage of potential respondents stopped completing the surveys 1) when encountering the first question, 2) when encountering a complex question grid, and 3) when asked to supply their email address. This suggests that some potential respondents have difficulty with the media and give up early in the process of completing the survey or when encountering complex questions. Others may be reluctant to give out personal information such as an email address. The logs were also merged with demographic data collected via the surveys. Somewhat surprisingly no patterns in failure to complete rates were found by gender, age or education level. In two of the surveys, people with lower income were found to have a higher rate of repeating screens of questions mainly due to improperly filling out questions.

As noted, due to their inherent advantages, most Internet surveying is now being done using HTML forms with potential respondents often contacted via email cover letters. While some developers still directly code these forms in HTML, there are dozens of HTML editors available, and they are becoming increasingly sophisticated and easy to use. There are two general methods of capturing the data entered into an HTML form. The form can be programmed to email the data back to a specified email address or captured by a program on the server called a common gateway interface (CGI) script. Using CGI scripts is more robust, offers more flexibility and is the far more commonly used method of capturing data. There are several HTML development packages that both provide HTML editing capabilities and automate the process of developing the CGI scripts necessary to capture data from HTML forms developed with the package. Two widely used examples of these packages are Microsoft’s FrontPage and Macromedia’s ColdFusion.

While these packages are general purpose Web development tools, there are also a growing number of software development systems designed specifically for Web-based surveying. Examples include Perseus’s Survey Solutions for the Web, Creative Research System’s The Survey System, and Survey Said™ Survey Software. These packages tend to offer additional features specific to survey research. Examples include as managing the distribution of email cover letters, built-in statistical analysis and reporting capabilities, and automatic tracking of people who have responded coupled with the ability of sending out follow-up email reminders to those who have yet to respond. Their HTML editors are also geared for survey form development, allowing them to simplify and streamline the process of developing and formatting the question response fields.

The author has developed a set of software tools that provides many of the complex Web survey administration functions included in Web surveying packages³. The software, Web Survey Mailer System (WSMS), is an integrated survey administration system that will send out personalized email cover letters, track which of the sampled people have completed the survey while keeping their responses anonymous and send out subsequent reminder emails to only those sampled people who have not responded to the survey. WSMS will block people who have not been sampled from accessing and responding to the survey and will keep respondents from submitting more than one set of survey responses. The system includes a customizable CGI script to capture the survey responses and place them in a tab-delimited ASCII database format that can easily be downloaded from the server and imported into a standard PC data base or statistical package. WSMS is written in PHP and uses the MySQL relational database to store information on the sampled people. Both PHP and MySQL are stable and powerful "open source" packages widely available on university and commercial Web servers and can be obtained free of charge in a variety of versions that will run on most common server operating systems. The WSMS scripts and documentation are available free-of-charge and can be downloaded from http://www.med-ed-online.org/rsoftware.htm#wsms

Internet surveys are clearly going to continue to grow in popularity as the problems of coverage bias and unfamiliarity with the Internet subside. For the foreseeable future there will be people who will lack Internet access either by choice or circumstance though this will be less and less of an issue. Additionally the tools for conducting Web-based surveys will continue to grow in sophistication and ease of use as will our knowledge on how best to employ this survey methodology. At present researchers should use this technique with caution in carefully chosen populations and with an eye to learning as much as possible about how to do it better.

1. In this paper we use the term "Internet survey" for both email and HTML form-based surveying while the term "Web-based survey" is reserved for HTML form-based surveys.

2. Cook, Heath, & Thompson (2000) included studies of both Web- and email-based surveys.

3. Detailed documentation for the Web Survey Mailer System is provided. However, installing and using these tools requires a good working knowledge of HTML and some background and understanding of server-based programming.

Best, S.,J., Krueger, B., Hubbard, C. & Smith, A. (2001) An assessment of the generalizability of internet surveys. Social Science Computer Review, 19, 131-145.

Cook, C, Heath, F, & Thompson, R. (2000) A meta-analysis of response rates in web or Internet-based surveys. Educational and Psychological Measurement, 60, 821-836.

Couper, M.P., Blair, J. & Triplett T (1999) A comparison of mail and e-mail for a survey of employees in federal statistical agencies. Journal of Official Statistics, 15, 39-56.

Crawford, S.D., Couper, M.P.& Lamias, M.J.. (2001) Web Surveys: Perception of burden. Social Science Computer Review, 19, 146-162.

Dillman, D.A, Tortora, R.D, Conrad, J. & Bowker D. Influence of plan vs. fancy design on response rates of Web surveys. Working paper available from http://survey.sesrc.wsu.edu/dillman/papers.htm [Accessed 6/01]

Jeavons A. Ethology and the Web: Observing respondent behavior in Web surveys. Proceedings of the Worldwide Internet Conference, Amsterdam: ESOMAR, 1998, available from http://w3.one.net/~andrewje/ethology.html [Accessed 7/01]

Kaye B.K. & Johnson T.J. (1999) .Research Methodology: Taming the Cyber Frontier. Social Science Computer Review, 17, 323-337.

Kittleson, M. (1997) Determining effective follow-up of e-mail surveys. American Journal of Health Behavior. 21, 193-196.

Medin, C., Roy, S. & Ann, T. (1999) World Wide Web versus mail surveys: A comparison and report.. Paper presentation at ANZMAC99 Conference, Marketing in the Third Millennium, Sydney, Australia, available from http://www.anzmac99.unsw.edu.au/anzmacfiles/papers.htm [Accessed 6/01]

Selwyn, N., Robson, K. (1998) Using e-mail as a research tool, Social Research Update, available from http://www.soc.surrey.ac.uk/sru/SRU21.html [Accessed 6/01]