Designing Surveys

1. Designing Surveys September 18, 2012 1:00 – 2:15 pm

2. Workshop objectives • By the end of this session you will: • Know the major strengths and weaknesses of alternative methods for measuring key public health variables using survey questionnaires • Be able to write clear, specific survey questions • Be able to independently design a questionnaire for a small-scale survey

3. Preparing for a Survey • Defining the survey purpose • Identifying survey recipients • Defining key concepts • Designing the survey instrument • Putting it all together (structure)

4. Why use a survey? • Target large amount of people • Use to describe, compare or explain • Can cover activities and behavior, knowledge, attitudes, preferences • Specific objectives, standardized and highly structured questions • Used to collect quantitative data – information that can be counted or measured

5. Strengths • Reach respondents in widely dispersed locations • Can be relatively low cost in time and money • Relatively easy to get information from people quickly • Standardized questions • Analysis can be straight-forward and responses pre-coded • Low pressure for respondents Limitations • Low response rate and consequent bias and confidence in results • Seeks information only by asking, can we trust what people say? e.g. issues with over-reporting • Question wording can have major effect on answers

6. Biases that must be understood • The selection of questions reflect the biases and perspectives of the researchers • The questions being posed often include inherent biases • Answers to survey questions reflect the perspective of the respondent • Multiple respondents may answer differently • The selected sample may not necessarily reflect the general population • Difficulty in getting “accurate” responses from different populations

7. Defining the Survey Purpose

8. Write a couple of sentences about the purpose of the survey- what is the general thing you want to find out? • Write down ideas about the kinds of questions you could ask • Brainstorm the data you could collect related to the purpose you have chosen • Write down the users for the questionnaire • Group your ideas into three or four key areas to investigate and write down what these are • Why will this be important to the person who wants the questionnaire written? Think about the overall purpose for your questionnaire and what data you want to get out of it Develop the purpose

9. Things to think about: Kind of information: What do you want to know? Is the information already available? What do you NEED to know vs. “would like to know” Who will complete the questionnaire - when? (think about readability, length, interest in the topic, etc)

10. Identifying Survey Recipients

11. Who do you want to find out information about? Who will you give the questionnaire to? This is the target population. This is called the sample.

12. Sampling COMBINE WITH PREVIOUS Good Sample Population Poor Sample

13. What is a Response rate? The proportion of people who respond: Example: If you distribute 50 questionnaires and you get 25 questionnaires back, your response rate is 50%. # that answered = response rate # you contacted • High response rate promotes confidence in results. • Lower response rate increases the likelihood of biased results.

14. Which type of survey should I choose?

15. 5 Alternative Interview Methods

16. 5 Alternative Interview Methods(continued)

17. Which type of survey should I choose? “It depends”… upon • What you want to know – how complex or sensitive the information is • Who the respondents are – their characteristics and which type of survey will be most appropriate • Your timeline • Available resources

18. Will using a survey be culturally appropriate? Literacy level Tradition of reading, writing Setting Not best choice for people with oral tradition Translation (demands more than just literal translation) How cultural traits affect response How to sequence the questions Pretesting the questionnaire when it may be viewed as intrusive Computer access and use if an electronic survey

19. Identifying characteristics of the respondents is important because: • Their educational backgrounds determine the most effective wording • Their ages and writing abilities will affect the type of question you use • Their potential interest in the topic affects what you need to do to ensure they respond • Physical and ability levels may affect how the questionnaire is administered

20. Defining Key Concepts

21. Define the variables to be collected • Write a detailed list of the information to be collected and the concepts to be measured in the study. Are you trying to identify: • Attitudes • Needs • Behavior • Demographics • Some combination of these concepts • Translate these concepts into variables that can be measured. • Define the role of each variable in the statistical analysis: • Predictor • Confounder • Outcome

22. Designing A Survey Instrument Creating and aggregating information about a large number of respondents

23. Study protocol and questionnaire Questionnaire Study protocol Definition of objectives Design questions questions must relate to objectives collect appropriate information for analyses Control and reading List of variables to collect Pilot test Analysis plan

24. Good questionnaires are difficult to construct; bad questionnaires are difficult to analyze. Questionnaire design Design individual questions: • Adopt questions used in other questionnaires • Adapt questions used in other questionnaires (search the literature) • Develop your own questions (may want to use focus groups or key informant interviews)

25. Review the literature • Review current literature to identify related surveys and data collection instruments that have measured concepts similar to those related to your study’s aims. • Saves development time and allows for comparison with other studies if used appropriately. • Proceed with caution if using only a subset of an existing questionnaire as this may change the meaning of the scores. Contact the authors of the questionnaire to determine if a smaller version of the instrument exists that has also been validated.

26. Questionnaire needs to be adapted to study population Know the respondents • language • education • occupation • ethnic group • sensitive issues

27. 5minute Break

28. Questions • Type of information • Type of questions • Types of response • Common wording problems • Problems with response • Validity • Reliability

29. Type of information • Knowledge - what people know What is the recommended interval between eye checks for patients with uncomplicated diabetes? • 6 months • 1 year • 2 years • Not sure

30. Type of information • Knowledge - what people know • Opinions, attitudes, beliefs, values - what people think about an issue What do you think are the major issues affecting general practice physicians at the moment? ___________________________________________________ ___________________________________________________

31. Type of information • Knowledge - what people know • Opinions, attitudes, beliefs, values - what people think about an issue • Behavior - what people do Have you developed a care plan for any of your patients? • Yes • No

32. Type of information • Knowledge - what people know • Opinions, attitudes, beliefs, values - what people think about an issue • Behavior - what people do • Attributes – who people are (characteristics) When did you graduate from college? _______

33. questions • Type of information • Type of questions • Types of response • Common wording problems • Problems with response • Validity • Reliability

34. Open and closed questions (from Oppenheim, 1992)

35. Closed-ended questions • Typically used in quantitative studies. • Assumption is researcher has knowledge to pre-specify response categories. • Data can be pre-coded and therefore in a form amenable for use with statistical packages (e.g., SPSS, SAS) – data capture therefore easier. • More difficult to design but simplifies analysis. • Used in studies involving large samples. • Limited range of response options.

36. Branching questions • Are used to direct respondents to answer the right questions as well as questions in the proper sequence • ‘Skips’ in telephone questionnaires more easily managed if these are computer-assisted • Consider including ‘not applicable’ category: In the past week, how often have you used MEDLINE: • Not at all • At least once • More than once • I do not have access to MEDLINE

37. Screening or filter questions • Are used to ensure respondents in the study are those that meet the pre-determined criteria of the target population. “Tonight we are talking with individuals who are 18 years of age or older and have 50 percent or more of the responsibility for healthcare decisions in your household. Are you that person?” __ Yes __ No

38. questions • Type of information • Type of questions • Types of response • Common wording problems • Problems with response • Validity • Reliability

39. Response – wording the answer • Wording the responseis as important as clear wording in the question. • Make the answer options clear, logical, comparable and mutually exclusive.

40. responses • Closed ended questions are usually followed by a set of responses • Limited choices of responses, lack of consistency in what a yes/no, agree/disagree response means Do you have trouble climbing stairs? • Attitudes and behaviors lie on a continuum To what extent do you experience difficulty when climbing stairs in your house? • None • A little • Quite a bit • A lot • I do not have stairs in my house

41. responses • Choose type of scale: • Nominal Are you: • Male • Female What is your marital status? • Single • Married • Widowed • Divorced • Separated • Other, specify __________

42. responses • Choose type of scale: • Nominal • Ordinal What is your current stage of cancer? • Localized • Regional • Metastatic What is the highest level of education you have reached? • Did not complete high school • High school diploma/GED • Some college • Bachelor’s degree • Post-graduate education

43. responses • Choose type of scale: • Nominal • Ordinal • Continuous • Numerical How severe was your pain after you were stung? (please circle) 1 2 3 4 5 6 7 Not painful at all Very painful • Analogue How severe is your pain (put the tick on the line) 0 10

44. How many times have you participated in the health fair? Poor spacing and logic Better 0−1 1−3 3−6 7−12 More than 12 times 0 times (never) 1−3 times 4−6 times 7- 9 times 10 or more times Vertical order of answer choice options

45. Exhaustive and mutually exclusive response options From which one of these sources did you first learn about the health clinic? • A friend • A neighbor • A relative • An advertisement • At school What is your age ? • 16-20 • 20-25 • 25-30 • 35-40 • Over 40? • Mutually exclusive: In the above example, I might have first learned about the health clinic through a friend who is also a neighbor – which would I check? Also, the choices include both sources and locations. • Exhaustive: make sure the options include all possible, important answers. Use ‘other’ fields.

46. Include both positive and negative sides in the question stem and all possible answer options Poor:Do you agree that our planning committee needs to meet once per month? • Agree • Disagree Better:Do you agree or disagree that our planning committee needs to meet once per month? • Agree • Disagree • No opinion Another option: How often do you think our planning committee should meet? • More than once a month • Once a month • Less than once a month

47. Rating scales

48. Five types of ordinal rating scales • Endorsement • Definitely true, true, don’t know, false, definitely false • Frequency • Always, very often, fairly often, sometimes, almost never, never • Intensity • None, very mild, mild, moderate, severe • Influence • Big problem, moderate problem, small problem, very small problem, no problem • Comparison • Much more than others, somewhat more than others, about the same as others, somewhat less than others, much less than others

49. Provide balanced scales with categories that are evenly spaced conceptually Poor Better Overall, how would you rate your experience as a nurse? • Excellent • Very good • Good • Fair • Poor Overall, how would you rate your experience as a nurse? • Very good • Good • Fair • Poor • Very poor Balance the scale with an equal number of positive and negative options.

50. Make sure the response scale matches the question Poor Better To what extent do you think the health leadership project helped you develop your leadership skills? • Excellent • Very good • Good • Fair • Poor To what extent do you think the health leadership project helped you develop your leadership skills? • Very great extent • Great extent • Some extent • Little extent • Very little extent