Survey Design and Analysis: Planning a Survey

1. Survey Design and Analysis:Planning a Survey Division of MH/DD/SAS Quality Management Team

2. How to plan a survey Development of Proposal Instrument Development and Survey Design Data Collection Post-collection Processing Analysis and Dissemination

3. Development of Proposal • Goal: identify a need for survey data and obtain resources for data collection • Key tasks: • Define questions to be answered by the survey • Review existing data and literature • Outline survey instrument • Set rough timetable for survey operations and analysis

4. Instrument Development and Survey Design • Goal: Develop survey tools to meet the objectives, given resources • Key tasks: • Develop Request for Proposals for survey vendor bids, if needed • Finalize survey timetable • Develop sampling procedures • Create and edit test survey instrument • Develop interviewer training materials • Conduct field test of survey • Evaluate and revise test instrument based on the field test • Develop final instrument • Work out a plan for analyzing the survey data

5. Designing a Questionnaire • The place to start in designing a questionnaire is with your data collection goals • What information do you need and from whom? • What pieces of specific information are needed to satisfy these objectives?

6. Types of Questions Open-ended Partially open ended Closed-ended Rating/Likert Scale

7. Use Appropriate Language DON’T: In the past 30 days, how often have you used Tina? DO: In the past 30 days, how often have you used crystal methamphetamines (Tina, speed, ice, glass)?

8. Use Simple Sentence Structures DON’T: In the past 30 days, when you used needles for injecting drugs, have you shared them with anyone else? DO: In the past 30 days, have you shared needles?

9. Avoid “Double-Barreled” Questions DON’T: Are you getting along better with your family and friends? DO: Are you getting along better with your family ? And Are you getting along better with your friends?

10. Avoid embedding assumptions or opinions DON’T: Have you ever inappropriately gone to an Emergency Department when you were in need of mental health services, but your regular provider was not available? DO: Have you ever gone to an Emergency Department when you were in need of mental health services, but your regular provider was not available?

11. Data Collection • Goal: Gather raw data in a timely and cost-effective manner • Key Tasks: • Conduct training • Field the survey • Data capture • Monitoring interviewers • Communication between vendor and sponsor to resolve operational issues

12. Post-Collection Processing • Goal: Generate accurate and organized final data set • Key Tasks: • Clean the data • Edit the data • Missing data imputation, if desired • Weighting, if desired • Generate data dictionary and code book • Contains data set description, variable names, and basic tabulations

13. Summary • Design questions carefully, paying attention to: • Question wording • Person-centered language, appropriate vocabulary level • Length of questions • Ensure questions are not too long and do not contain multiple concepts. • Length of questionnaire • If long, you may not get meaningful data for questions at the end. • Order of questions • Appearance of questionnaire

14. How to “plan in” quality • Plan in quality to keep respondent mistakes and biases to a minimum. • Make sure there are mechanisms for quality checks at every point. • Make sure sample is selected according to specifications. • Pretest questions to ensure they are getting the data you need. • If using interviewers, make sure they are appropriately trained and understand their task. • Ensure survey answers are coded correctly. • Ensure computer programs for data analysis are working properly.

15. Judging the quality of a survey • There are many issues that can impact the quality of survey data. • This section describes potential issues and remedies for common problems.

16. Survey non-response and measurement • Non-response • Potential respondent can not or will not participate in the survey or answer specific questions • Measurement error • Bias or error when surveys do not measure what they intend to • Example: Measuring consumer satisfaction with case manager when you are trying to measure satisfaction of services in their entirety

17. How do problems affect survey results? • Bias • Nonresponse bias is the bias that results when respondents differ in meaningful ways from nonrespondents. • Variance • Less predictable effect that may cause projections to be higher one time but lower the next time

18. Frequent causes of non-response error • Refusals • Individuals are approached to participate in a survey but refuse to do so. Refusals may result from apathy, fear of invasion of privacy or any number of reasons. Some refusals are partial, where the respondent will answer some questions but not all. • Unable to answer • This includes persons who are unable to communicate for a variety of reasons. Many times a caregiver or family member can serve as a proxy for these individuals. • Not Founds • This includes individuals who have moved and left no forwarding address, have no permanent address, are deceased, or are not contacted due to error in the survey procedures.

19. How to handle non-response errors • Generalize to the respondents only. In stating that results are accurate for those surveyed, you can avoid making incorrect inferences about the larger population. • Assume there is no response bias and generalize to the population. If you know the population well and perceive the results to be reasonable, this strategy may be reasonable. • Recontact nonrespondents. Finding out why people did not respond can help determine the extent of response bias. If none is apparent then generalization to the population can be justified.

20. How to handle non-response errors • Compare data in hand on respondents and nonrespondents. If data, e.g., sex, age, race, is available, the composition of respondents can be compared with that of nonrespondents to see if there are any differences. The presence of differences indicates response bias and that caution is necessary in making inferences. • Increase contact efforts. Although this is probably the most costly and time-consuming strategy, obtaining the highest response rate possible is the best way to reduce response bias. • Throw away the data. If you feel the survey results are invalid, a logical response is not to use it. For many reasons, this strategy is not often used.

21. Frequent causes of measurement error • Failure to identify the target population • Surveying individuals who have only been briefly engaged in service • Questionnaire design • Interviewer bias • Respondent bias • Processing errors • Misinterpretation of results

22. Frequent causes of measurement error • Failure to identify the target population • Questionnaire design • Interviewer bias • Respondent bias • Processing errors • Misinterpretation of results

23. How to prevent measurement errors • Careful selection of the time and place the survey is conducted • Using an up-to-date, accurate sample framework • Careful questionnaire design • Providing thorough training for interviewers and processing staff • Being aware of all the factors affecting the topics that are being addressed by the survey