Tools for Research and Inquiry: Improved Survey Design

1. Tools for Research and Inquiry: Improved Survey Design

2. Brief overview of surveys Constitute one of the most important and common tools in applied social science research Comprised of a predetermined set of questions designed to collect information from a group of individuals Implemented by personally/orally (face-to-face or by phone) or self-administered (by mail or internet)

3. Planning a survey What are my research questions? What do I want to learn/understand? What information do I need to collect to answer my questions and where is that information located? How does the survey fit into my overall research design? What specific research questions do I hope to address using a survey? Will the survey be cross-sectional (point in time), or will it need to be longitudinal to collect the information needed to respond to my questions? How do I hope to use what I learn from the survey? What type of sample will be needed to address the research questions and enable me to use the results in the way that I would like?

4. Planning a survey How can I best contact the target population? When am I most likely to reach my target population and when are they most likely to respond? What type of administration will elicit the largest quantity and highest quality of responses? How do I plan to analyze the data I collect? If needed, how will I handle data entry and data cleaning? What is my budget for the survey�design, implementation, analysis and reporting? Have previously conducted surveys collected relevant data that are available for secondary analysis? What is my plan for pre-testing the survey? What are my plans for using, disseminating, and reporting the survey results?

5. Pretesting the survey Ask pre-testers to set aside sufficient time to do the survey in one session and to record the time when they begin and when they complete the survey. Put yourself in the place of your respondents--how much time would you be willing to spend on your response? Ask them to make a note of any questions they found confusing, or any questions for which none of the response options applied to them. Ask how they felt about the flow of the questions, if they were able to follow the skip patterns (if any.) Ask about any general reactions they had to individual questions or the survey overall.

6. Designing questions and surveys Question type Levels of measurement Item content/wording Item order Format and response options Response burden

7. Implementation considerations Provide advance notice Use multiple follow-ups/reminders Make each contact unique Explain the research purpose, how they were selected Emphasize why it is important they respond and how their confidentiality will be protected Provide an estimate of how long it will take and information about any incentive Describe how the results will be used and offer a summary, if feasible

8. Potential sources of error/bias Coverage Sampling Item bias/misinterpretation Data misinterpretation Data entry Low response rates /Non-response

9. Comparison of Survey Implementation Modes

10. Relatively inexpensive method, especially if self-administered Can be administered remotely, though remote administration does not provide respondents an opportunity to ask clarifying questions Large and geographically dispersed samples are feasible, increasing the possibility of statistically significant results even when analyzing multiple variables Careful attention to wording and format can make surveys highly reliable (At the same time, poorly designed, inadequately pretested surveys yield unreliable data.) Standardized questions increase the precision and reliability of measurement by enforcing uniform categories on respondents, but this makes surveys inflexible tools Surveys can be weak on validity, since the format often forces respondents into dichotomous categories Yes/No, Agree/Disagree, etc. Print and electronic surveys may not be well suited to some populations, e.g., low literacy, speakers of English as a second language, homeless or transient populations without access to mail or internet Summary of Survey Strengths & Limitations

11. References Colorado State University (1993-2010) Retrieved from: http://writing.colostate.edu/guides/research/survey/ Converse, Jean M., & Presser, Stanley. (1986). Survey questions: Handcrafting the standardized questionnaire (Vol. 63). Newbury Park: Sage Publications, Inc. Cook, Coleen, Heath, Fred, & Thompson, Russell L.. (2000). A meta-analysis of response rates in web- or internet-based surveys. Educational Psychology and Measurement, 60, 821-836. Couper, Mick P., Traugott, Michael W., & Lamias, Mark J. (2001). Web survey design and administration. The Public Opinion Quarterly, 65(2), 230-253. Dillman, & A., D. (2000). Mail and internet surveys: The tailored design method (Second ed.). New York: John Wiley and Sons, Inc. Fan, Weimiao & Zheng, Yan. (2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior, 26, 132-139. Garson, G. David (2009) http://faculty.chass.ncsu.edu/garson/PA765/survey.htm Greene, Jennifer C. (1991). HSS 690 Course Materials. Kaplowitz, Michael D., Hadlock, Timothy D., & Levine, Ralph. (2004). A comparison of web and mail survey response rates. Public Opinion Quarterly, 68(1), 94-101. Schonlau, M., Fricker, R. D., Jr., & Elliott, M. N. (2001). Conducting research surveys via e-mail and the web (pp. 118). Retrieved from http://www.rand.org/publications/MR/MR1480/ Sheehan, & Kim. (2001). E-mail survey response rates: A review. Journal of Computer-Mediated Communication, 6(2).