Questionnaire Design & Formatting in Qualtrics Day 2

1. Questionnaire Design & Formatting in QualtricsDay 2 Nathan R. Jones, PhD Senior Project Director University of Wisconsin Survey Center nrjones@ssc.wisc.edu

2. Schedule • Day 1 - Overview • Day 2 - Questionnaire development • Day 3 - Data collection

3. Principal parts of a survey • Study Design • Sample Design • Mode of administration • Instrument development • Pretesting, testing, IRB • Data collection • Data preparation - weighting, processing, delivery • Documentation • Analysis • Budget

4. A Survey From a Process Perspective (Groves et al. 2004)

5. Measurement: Terms and definitions Conceptualization • Defining, formulating, and clarifying concepts by • specifying dimensions • selecting indicators Operationalization • Exact procedures used to obtain a measurement of a construct

6. Example • How do we measure “health care use”? • Specifying dimensions:

7. Example • How do we measure “health care use”? • Specifying dimensions: • Who qualifies as a health care provider? • What reasons for use should be included? (e.g., only major problems or all)? • Include self only or all family members? • Include measure of frequency? • What is the reference period?

8. Example • Operationalization of a construct During the past 12 months, how many times have you seen a doctor or other health care professional about your own health at a doctor’s office, a clinic, or some other place? (Do not include times you were hospitalized overnight, visits to hospital emergency rooms, home visits, or telephone calls)

9. Evaluating survey questions • Reliability • Repeated measurements with the same measurement instrument gives the same value • Validity • Instrument measures construct it is supposed to measure

10. Improving standardization through question design • Goal of standardization is to reduce measurement errors • In interviewer-administered surveys, interviewers must be able to… • read questions exactly as worded • record answers completely and accurately • provide neutral clarifications of survey concepts • In self-administered surveys, respondents must be able to… • read questions exactly as worded • record their answers completely and accurately • understand the meaning of words and phrases as investigators intended

11. Instrument complexity Complexity of the instrument • Are there complicated skip patterns? • Will there be more than one version of the instrument or more than one instrument? • Will you need to know about answers to prior questions? • Will household members need to be “rostered?” • Are there grids, for example, that ask a series of questions about each person in the household?

12. Writing questions • Good question must… • yield responses that are reliable and valid • promote easy and accurate cognitive processing • be understood by all respondents in the same way • be understood by all respondents as intended by the question writer • Respondents must be willing and able to answer • Interviewers must be able to read the question accurately and consistently

13. Problematic Questions: Agree/Disagree How strongly do you disagree or agree with the following statements: Strongly disagree, disagree, agree, or strongly agree?

14. Example Changing Agree/Disagree to construct specific question How strongly do you disagree or agree with the following statement: - I am satisfied with this program. [SA] [A] [D] [SD]

15. Example How strongly do you disagree or agree with the following statement: - I am satisfied with this program. [SA] [A] [D] [SD] How satisfied are you with this program? • Extremely satisfied • Very satisfied • Somewhat satisfied • Neutral • Somewhat unsatisfied • Very unsatisfied • Extremely unsatisfied

16. Example: Response dimension mismatch I would be treated respectfully if I went to the Prefecture Hospital for a medical problem. Is that statement: Very untrue Somewhat untrue Both true and false Somewhat true Very true

17. Example: Clarifying the response dimension How often are you treated respectfully when you visit the Prefecture Hospital for a health problem? Never Sometimes Most of the time All the time

18. Example: Clarifying the reference period In the last 3 months, how often were you treated respectfully when you visited the Prefecture Hospital for a health problem? Never Sometimes Most of the time All the time

19. Self-administered questionnaires • Appealing cover and graphics • Block question text • Overhang question numbers to left 1) Question • Response 1 • Response 2 • Distinguish questions from instructions • Keep it clean • No parentheses, slashes, or abbreviations in question text • Use conventions consistently • No extraneous type • Leave white space • No more than four variations in type or font • Include contact information

20. Analytic Vocabulary: Revision: - occupants of this household people who live here Avoid analytic vocabulary Analytic vocabulary: language used by the analyst such as technical terms and academic jargon. - your responses to this questionnaire your answers - work-related employment issues job concerns

21. Example Problematic Question: How strong is your fertility motivation: Not at all, a little, somewhat, quite, very, or extremely?

22. Example Problematic Question: How strong is your fertility motivation: Not at all, a little, somewhat, quite, very, or extremely? Revision: How sure are you that you want another child: not at all sure, a little sure, somewhat sure, quite sure, very sure, or extremely sure?

23. Avoid negative constructions A negative construction asks the respondent to consider the negation of a target object or poses a question with a double-negative. Problematic Question: Do you agree or disagree that children should not have to take care of their parents when the parents become old?

24. Avoid negative constructions A negative construction asks the respondent to consider the negation of a target object or poses a question with a double-negative. Problematic Question: Do you agree or disagree that children should not have to take care of their parents when the parents become old? • Revision: • Do you agree or disagree that children should take care of their parents when the parents become old?

25. Avoid leading words or phrases Leading, biased, or emotionally loaded words or phrases provide a point of view or orientation toward an object. Problematic Questions: Do you favor or oppose President Jones’s humanitarian policy in Somalia? Do you favor or oppose President Jones’s harmful policy in Somalia?

26. Avoid leading words or phrases Leading, biased, or emotionally loaded words provide a point of view or orientation toward an object. Problematic Questions: Do you favor or oppose President Jones’s humanitarian policy in Somalia? Do you favor or oppose President Jones’s harmful policy in Somalia? • Revision: • Do you favor or oppose President Jones’s policy in Somalia?

27. Effects of small question wording changes • Do you think the United States should forbidspeeches against democracy? (n=1475) • Yes (forbid): 21% • No (no forbid): 79 • Do you think the United States should allowspeeches against democracy? (n=1375) • No (not allow): 48% • Yes (allow): 52% • Difference: 48-21=27%; X2=223.2, p<.001 Source: Schuman and Presser 1981

28. Avoid compound or complex question structures A compound or complex question includes two or more clauses. Problematic Question: In the last 12 months, did you ever reduce the size of your meals or skip meals because there wasn’t enough money for food?

29. Avoid compound or complex question structures A compound or complex question includes two or more clauses. Problematic Question: In the last 12 months, did you ever reduce the size of your meals or skip meals because there wasn’t enough money for food? Revision: In the last 12 months, did you ever reduce the size of your meals because there wasn’t enough money for food? In the last 12 months, did you ever skip meals because there wasn’t enough money for food?

30. Exhaustive and mutually exclusive Problematic Question: How did you first learn about the earthquake in Turkey? RADIO TELEVISION SOMEONE AT WORK WHILE AT HOME WHILE TRAVELING TO WORK Exhaustivecategoriesallow all respondents to be classified Mutually exclusive categories do not overlap so each respondent can be classified in only one category

31. Exhaustive and mutually exclusive From which one of these sources did you first learn about the earthquake in Turkey? RADIO TELEVISION ANOTHER PERSON Where were you when you first heard about it? AT WORK AT HOME TRAVELING TO WORK SOMEWHERE ELSE

32. Exercise: Question 1 Problematic Question: When did you last see a doctor? MONTH _____ YEAR _____

33. Exercise: Question 1 Problematic Question: When did you last see a doctor? MONTH _____ YEAR _____ Revision: In what month and year was your most recent visit to a medical doctor about your own health? MONTH _____ YEAR _____

34. Exercise: Question 2 Problematic Question: Would you say you are physically more active, less active, or about as active as other persons?

35. Exercise: Question 2 Problematic Question: Would you say you are physically more active, less active, or about as active as other persons? Revision: Thinking about your physical activity during the last 30 days, would you say you are more active, less active, or about as active as other persons your age?

36. Exercise: Question 3 Problematic Question: How often do you use a seatbelt when you drive a car? Always, nearly always, sometimes, never? Always Nearly always Sometimes Never

37. Exercise: Question 3 Revision: 1A. During the past 30 days, that is since (DATE), did you drive a car, van, or truck? YES NO 1B. Considering the times that you drove during the past 30 days, how often did you use a seat belt? Would you say... Never Less than half the times you drove About half the times you drove More than half the times you drove Every time you drove

38. Exercise: Question 4 Problematic Question: When you were growing up, did you live with anyone in your household who smoked? Y/N

39. Exercise: Question 5 Problematic Question: When you were growing up, did you live with anyone in your household who smoked? Revision: When you were growing up, that is during your first 16 years, which of the following people lived in your household and smoked? Father YES NO Mother YES NO Anyone else YES NO

40. Pay attention during testing • Flow of instrument • Time • Debugging

41. General recommendations • What are some strategies for effectively wording questions? • Ask about what you need to know (related to your research question) • Ask questions that are easy to answer (low cognitive burden) • Make sure answers match the question (response dimension) • Ask about one item/concept/thing at a time (avoid double barreled) • What types of questions make a questionnaire useful when analyzing the data, especially when using statistics? • START with an analysis plan • Are you going to compare your data with another study? • How will you use every question? • Think about response categories - Continuous, categorical, open-ended, check one, check all that apply?

42. General recommendations • How to construct questions and responses to be as unbiased as possible? • Avoid leading statements • Test question and response category ordering • Borrow from validated instruments • How do I make it long enough but not too long? • Test, test, test • Adjust incentives • How do I take foreign languages into account when writing a questionnaire? • Translate, then independently back-translate • Test, test, test

43. Questions? http://bit.ly/1ZSwk6J

44. Thank you! Nathan R. Jones Senior Project Director University of Wisconsin Survey Center nrjones@ssc.wisc.edu