Mode Consistency Guidelines: Asking Questions in Multimode Surveys

1. Mode Consistency Guidelines:Asking Questions in Multimode Surveys Paper Presented at 2007 FedCASIC Jennifer Hunter Childs, Theresa DeMaio, Eleanor Gerber, Joan Hill, Elizabeth Martin, and Courtney Reiser U.S. Census Bureau

2. The Impetus • Multiple modes of data collection for 2010 Census and American Community Survey (ACS): • Self-administered paper questionnaires • Computer-assisted telephone interviews • Computer-assisted personal visit interviews • Paper-administered personal visit interviews

3. The Impetus (cont.) • Variation between past instruments in: • Question wording • Response Task • Layout

4. The Working Group • Census Bureau Staff • Outside Experts

5. The Guidelines • 30 guidelines • Created for decennial demographic questions • Can be used as a model for multimode surveys

6. The Central Principle: Universal Presentation • All respondents should be presented with the same question and response categories, regardless of mode. That is, the meaning and intent of the question and response options must be consistent.

7. Universal Presentation • What this DOES NOT mean: • Everything should be identical • What this DOES mean: • meaning and intent must be consistent • Goal = collect equivalent information regardless of mode

8. Intent of the Guidelines • Realistic enough to be implemented • Considering interviewer and respondent burden • Allow flexibility for well-designed instruments to exploit the advantages of a particular mode

9. 8 Aspects of Survey Design • Question wording and instructions (1-8) • Use of examples (9-11) • Response categories (12-15) • Formatting of answer spaces (16-17) • Visual design elements (18-20) • Question order and grouping (21-23) • Flashcards (24-26) • Prompts and help (27-30)

10. Question Wording and Instructions Guideline 1 – Part 1 • Every effort should be made to maintain the same wording of questions across survey modes.

11. Guideline 1: Discussion • Minor variations • Can lead to differences in how questions are interpreted • Can cause large differences in data

12. Guideline 1 – Part 2 • When variant wording for a particular mode is proposed, the decision to change wording should be based upon evidence that a difference in wording is more likely than the same wording to produce consistent meaning and equivalent answers.

13. Guideline 1: Discussion • If the question cannot be adapted easily: • Look for a common wording that can be used in all modes. • Consider an exception for the problematic mode.

14. Guideline 1: Example • Self-Administered • Interviewer-Administered

15. Guideline 1: Discussion • Sources of Evidence for a Change: • Qualitative testing • Experimental field testing • Survey methodological literature • Mixed-mode surveys or censuses

16. Guideline 1: Discussion • Standards of evidence should vary according to the performance of a question: • Performing poorly • Working well • All changes should be pretested!

17. Question Wording and Instructions:Guideline 2 • Preambles and instructions on the answering process may require different wording to be appropriate to the individual mode.

18. Guideline 2: Discussion & Example • Preambles and instructions often need to be varied to facilitate responding. • Self-administered • Interviewer-administered

19. Question Wording and InstructionsGuideline 4 • The elements of a question may be reordered if necessary to ensure that respondents are exposed to all essential information in interviewer-administered modes.

20. Guideline 4: Discussion • Sometimes in a self-administered questionnaire, information is placed after the question mark. • In interviewer-administered surveys, this causes respondent break-ins.

21. Guideline 4: Discussion • Response options, examples and/or instructions: • May be incorporated as part of the question • May be presented on a flashcard • All changes should be pretested!

22. Guideline 4: Example • Self-Administered • Interviewer-Administered • Does Jane ever live or stay someplace else to attend college, to be closer to work, while in the military. . .

23. Question Wording and Instructions:Guideline 5 • Substantive instructions and explanations should be consistent across modes.

24. Guideline 5: Discussion & Example • Explanations to respondents should be consistent across modes, unless different explanations apply. • Example: Why we request a telephone number

25. Guideline 5: Discussion & Example • Substantive preambles should be conveyed consistently across modes. • Example - “We need to count every person living in the United States on April 1, 2010.”

26. Question Wording and InstructionsGuideline 6 • The underlying response task posed by a question should be consistent across modes, even if the implementation differs.

27. Guideline 6: Discussion • Asking respondents to perform the same task across modes maximizes the consistency of responses.

28. Guideline 6: Example Self-Administered Interviewer-Administered

29. Guideline 6: Discussion • Branching is permitted. • Branching questions can improve administration and obtain comparable data • But, branching may alter the meaning • All changes should be pretested!

30. Guideline 6: Branching Example

31. Guideline 6: Branching Example • Please choose one or more categories that best describe your race. • [1] White [5] Native Hawaiian • [2] Black, African-American, or Negro [6] Other Pacific Islander** • [3] American Indian or Alaska Native [7] Or, some other race • [4] Asian* • *To what Asian group do you belong? Are you: • [1] Asian Indian [4] Japanese [7]Or, some other Asian group? • [2] Chinese [5] Korean • [3] Filipino [6] Vietnamese • **To what Pacific Islander group do you belong? Are you: • [1] Guamanian or Chamorro [2] Samoan • [3] Or, some other Pacific Islander group?

32. Guideline 6: Discussion • “If not obvious, ask” should not be used. • It is acceptable to instruct interviewers to “Ask or verify.” • It is permissible to use automated instruments to perform calculations.

33. Use of ExamplesGuideline 9 • The same examples, in the same order, should be presented in a question in all modes.

34. Guideline 9: Discussion • Examples help interpretation of question by indicating intended type of response. • A standard context is necessary. • Research should support selection of a short list of examples that can be readily administered in all modes.

35. Guideline 9: Examples

36. Use of Examples:Guideline 10 • Placement and presentation of examples may vary by mode, consistent with the principle of universal presentation.

37. Guideline 10: Discussion & Examples • Optimal placement of examples varies by mode. • Self-administered questionnaire • Interviewer-administered questionnaire

38. Response Categories:Guideline 13 • For closed questions, instruments in all modes should be designed and administered to expose respondents to the response categories, in the same order, and to present response categories with the question.

39. Guideline 13: Discussion • Results are not comparable if respondents select from different response categories. • Response categories provide context. • Reducing the number and length of categories may help respondents. • Branching is a permissible adaptation.

40. Guideline 13: Example

41. Wrap-up • Work in progress • Used to develop 2008 Census Dress Rehearsal and 2008 American Community Survey specifications • Need for training

42. Future Plans • Revise the Guidelines as necessary after implementing them in 2008 • Expand the scope of the Guidelines to include other surveys at the Census Bureau • Conduct research in areas suggested by this review of literature