1. Cognitive Testing: Washington Group Questions Kristen Miller, Ph.D. Question Design Research Lab National Center for Health Statistics

2. Cognitive Testing Objectives • Identify potential response errors related to question design • Understand why and how errors might occur • Differing interpretations • Memory difficulty • Assessment errors • Insufficient response categories

3. Cognitive Testing Objectives • Identify socio-cultural factors that might impact the question response • Level of education • Socio-economic status • Cultural beliefs • Racial, ethnic, gender identities • Access to knowledge and resources

4. Traditional Cognitive Interviews are Semi-structured • Core Question- interview is organized by the questions that are being tested • Probe Questions- open-ended, spontaneous, not pre-scripted, based on the information that the respondent provides

5. Traditional Cognitive Interviewing • Data collected are qualitative • Qualitative methodologies are used to analyze data • Small samples: approx. 10 – 30 participants

6. Washington Group Testing: Obstacles • Little or no training • No understanding of cognitive methods • Few resources (funding, technology) • Multiple languages • Multiple cultures • Multiple levels of development

7. Protocol Requirements • Structured interview guide • Very simple instructions • Few skip patterns • Quantitative data and analysis

8. Cognitive Interview Protocol Designed to illustrate: • Administration ease • Interpretations • Factors considered • Degree of consistency with physical abilities

9. Cognitive Interview Protocol • Demographic Section: Country, gender, age, SES • Question Testing Sections (Self-report and Proxy-report) • Core Question • Interviewer Coding • Open-ended Follow-up Probe • Cognitive Follow-up Probe • Functioning Follow-up Probe • Health Questions • Subjective health • Chronic condition list

10. Interviewer Code: Did the respondent… • need you to repeat any part of the question? Yes/No • have any difficulty using the response options? Yes/No • ask for clarification or qualify their answer? Yes/No

11. Open-ended Follow-up Probe • Why did you answer that way?

12. Cognitive Follow-up Probes: • How often they had difficulty • The amount of effort they needed to put into the activity • The degree of concern or worry they had regarding the difficulty

13. Functioning Follow-up Probes • Presence of a condition • Use of an assistive device • Degree of difficulty doing related activities

14. Health Status • General health question • Why did you answer that way? • Health conditions

15. Interviewer Debriefing • Details about the interview and interview setting • Insight into the quality of the information

16. Analysis Plans • Patterns in the response process • Interpretation • Judgment • Patterns of response error • Across countries and demographic variables

17. Design & Testing of Cognitive Protocol • Extensive testing in the United States • Input from QUEST • Pilot test in Mexico

18. Pilot Findings • Strengths • Relevant findings • Few resources needed • Concerns • Interviewer training (more than expected) • Protocol translations (critical and will differ by country)