1 / 42

Mode Consistency Guidelines: Asking Questions in Multimode Surveys

Mode Consistency Guidelines: Asking Questions in Multimode Surveys. Paper Presented at 2007 FedCASIC Jennifer Hunter Childs, Theresa DeMaio, Eleanor Gerber, Joan Hill, Elizabeth Martin, and Courtney Reiser U.S. Census Bureau. The Impetus.

carl
Télécharger la présentation

Mode Consistency Guidelines: Asking Questions in Multimode Surveys

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Mode Consistency Guidelines:Asking Questions in Multimode Surveys Paper Presented at 2007 FedCASIC Jennifer Hunter Childs, Theresa DeMaio, Eleanor Gerber, Joan Hill, Elizabeth Martin, and Courtney Reiser U.S. Census Bureau

  2. The Impetus • Multiple modes of data collection for 2010 Census and American Community Survey (ACS): • Self-administered paper questionnaires • Computer-assisted telephone interviews • Computer-assisted personal visit interviews • Paper-administered personal visit interviews

  3. The Impetus (cont.) • Variation between past instruments in: • Question wording • Response Task • Layout

  4. The Working Group • Census Bureau Staff • Outside Experts

  5. The Guidelines • 30 guidelines • Created for decennial demographic questions • Can be used as a model for multimode surveys

  6. The Central Principle: Universal Presentation • All respondents should be presented with the same question and response categories, regardless of mode. That is, the meaning and intent of the question and response options must be consistent.

  7. Universal Presentation • What this DOES NOT mean: • Everything should be identical • What this DOES mean: • meaning and intent must be consistent • Goal = collect equivalent information regardless of mode

  8. Intent of the Guidelines • Realistic enough to be implemented • Considering interviewer and respondent burden • Allow flexibility for well-designed instruments to exploit the advantages of a particular mode

  9. 8 Aspects of Survey Design • Question wording and instructions (1-8) • Use of examples (9-11) • Response categories (12-15) • Formatting of answer spaces (16-17) • Visual design elements (18-20) • Question order and grouping (21-23) • Flashcards (24-26) • Prompts and help (27-30)

  10. Question Wording and Instructions Guideline 1 – Part 1 • Every effort should be made to maintain the same wording of questions across survey modes.

  11. Guideline 1: Discussion • Minor variations • Can lead to differences in how questions are interpreted • Can cause large differences in data

  12. Guideline 1 – Part 2 • When variant wording for a particular mode is proposed, the decision to change wording should be based upon evidence that a difference in wording is more likely than the same wording to produce consistent meaning and equivalent answers.

  13. Guideline 1: Discussion • If the question cannot be adapted easily: • Look for a common wording that can be used in all modes. • Consider an exception for the problematic mode.

  14. Guideline 1: Example • Self-Administered • Interviewer-Administered

  15. Guideline 1: Discussion • Sources of Evidence for a Change: • Qualitative testing • Experimental field testing • Survey methodological literature • Mixed-mode surveys or censuses

  16. Guideline 1: Discussion • Standards of evidence should vary according to the performance of a question: • Performing poorly • Working well • All changes should be pretested!

  17. Question Wording and Instructions:Guideline 2 • Preambles and instructions on the answering process may require different wording to be appropriate to the individual mode.

  18. Guideline 2: Discussion & Example • Preambles and instructions often need to be varied to facilitate responding. • Self-administered • Interviewer-administered

  19. Question Wording and InstructionsGuideline 4 • The elements of a question may be reordered if necessary to ensure that respondents are exposed to all essential information in interviewer-administered modes.

  20. Guideline 4: Discussion • Sometimes in a self-administered questionnaire, information is placed after the question mark. • In interviewer-administered surveys, this causes respondent break-ins.

  21. Guideline 4: Discussion • Response options, examples and/or instructions: • May be incorporated as part of the question • May be presented on a flashcard • All changes should be pretested!

  22. Guideline 4: Example • Self-Administered • Interviewer-Administered • Does Jane ever live or stay someplace else to attend college, to be closer to work, while in the military. . .

  23. Question Wording and Instructions:Guideline 5 • Substantive instructions and explanations should be consistent across modes.

  24. Guideline 5: Discussion & Example • Explanations to respondents should be consistent across modes, unless different explanations apply. • Example: Why we request a telephone number

  25. Guideline 5: Discussion & Example • Substantive preambles should be conveyed consistently across modes. • Example - “We need to count every person living in the United States on April 1, 2010.”

  26. Question Wording and InstructionsGuideline 6 • The underlying response task posed by a question should be consistent across modes, even if the implementation differs.

  27. Guideline 6: Discussion • Asking respondents to perform the same task across modes maximizes the consistency of responses.

  28. Guideline 6: Example Self-Administered Interviewer-Administered

  29. Guideline 6: Discussion • Branching is permitted. • Branching questions can improve administration and obtain comparable data • But, branching may alter the meaning • All changes should be pretested!

  30. Guideline 6: Branching Example

  31. Guideline 6: Branching Example • Please choose one or more categories that best describe your race. • [1] White [5] Native Hawaiian • [2] Black, African-American, or Negro [6] Other Pacific Islander** • [3] American Indian or Alaska Native [7] Or, some other race • [4] Asian* • *To what Asian group do you belong? Are you: • [1] Asian Indian [4] Japanese [7]Or, some other Asian group? • [2] Chinese [5] Korean • [3] Filipino [6] Vietnamese • **To what Pacific Islander group do you belong? Are you: • [1] Guamanian or Chamorro [2] Samoan • [3] Or, some other Pacific Islander group?

  32. Guideline 6: Discussion • “If not obvious, ask” should not be used. • It is acceptable to instruct interviewers to “Ask or verify.” • It is permissible to use automated instruments to perform calculations.

  33. Use of ExamplesGuideline 9 • The same examples, in the same order, should be presented in a question in all modes.

  34. Guideline 9: Discussion • Examples help interpretation of question by indicating intended type of response. • A standard context is necessary. • Research should support selection of a short list of examples that can be readily administered in all modes.

  35. Guideline 9: Examples

  36. Use of Examples:Guideline 10 • Placement and presentation of examples may vary by mode, consistent with the principle of universal presentation.

  37. Guideline 10: Discussion & Examples • Optimal placement of examples varies by mode. • Self-administered questionnaire • Interviewer-administered questionnaire

  38. Response Categories:Guideline 13 • For closed questions, instruments in all modes should be designed and administered to expose respondents to the response categories, in the same order, and to present response categories with the question.

  39. Guideline 13: Discussion • Results are not comparable if respondents select from different response categories. • Response categories provide context. • Reducing the number and length of categories may help respondents. • Branching is a permissible adaptation.

  40. Guideline 13: Example

  41. Wrap-up • Work in progress • Used to develop 2008 Census Dress Rehearsal and 2008 American Community Survey specifications • Need for training

  42. Future Plans • Revise the Guidelines as necessary after implementing them in 2008 • Expand the scope of the Guidelines to include other surveys at the Census Bureau • Conduct research in areas suggested by this review of literature

More Related