
Cognitive Testing: Washington Group Questions Kristen Miller, Ph.D. Question Design Research Lab National Center for Health Statistics
Cognitive Testing Objectives • Identify potential response errors related to question design • Understand why and how errors might occur • Differing interpretations • Memory difficulty • Assessment errors • Insufficient response categories
Cognitive Testing Objectives • Identify socio-cultural factors that might impact the question response • Level of education • Socio-economic status • Cultural beliefs • Racial, ethnic, gender identities • Access to knowledge and resources
Traditional Cognitive Interviews are Semi-structured • Core Question- interview is organized by the questions that are being tested • Probe Questions- open-ended, spontaneous, not pre-scripted, based on the information that the respondent provides
Traditional Cognitive Interviewing • Data collected are qualitative • Qualitative methodologies are used to analyze data • Small samples: approx. 10 – 30 participants
Washington Group Testing: Obstacles • Little or no training • No understanding of cognitive methods • Few resources (funding, technology) • Multiple languages • Multiple cultures • Multiple levels of development
Protocol Requirements • Structured interview guide • Very simple instructions • Few skip patterns • Quantitative data and analysis
Cognitive Interview Protocol Designed to illustrate: • Administration ease • Interpretations • Factors considered • Degree of consistency with physical abilities
Cognitive Interview Protocol • Demographic Section: Country, gender, age, SES • Question Testing Sections (Self-report and Proxy-report) • Core Question • Interviewer Coding • Open-ended Follow-up Probe • Cognitive Follow-up Probe • Functioning Follow-up Probe • Health Questions • Subjective health • Chronic condition list
Interviewer Code: Did the respondent… • need you to repeat any part of the question? Yes/No • have any difficulty using the response options? Yes/No • ask for clarification or qualify their answer? Yes/No
Open-ended Follow-up Probe • Why did you answer that way?
Cognitive Follow-up Probes: • How often they had difficulty • The amount of effort they needed to put into the activity • The degree of concern or worry they had regarding the difficulty
Functioning Follow-up Probes • Presence of a condition • Use of an assistive device • Degree of difficulty doing related activities
Health Status • General health question • Why did you answer that way? • Health conditions
Interviewer Debriefing • Details about the interview and interview setting • Insight into the quality of the information
Analysis Plans • Patterns in the response process • Interpretation • Judgment • Patterns of response error • Across countries and demographic variables
Design & Testing of Cognitive Protocol • Extensive testing in the United States • Input from QUEST • Pilot test in Mexico
Pilot Findings • Strengths • Relevant findings • Few resources needed • Concerns • Interviewer training (more than expected) • Protocol translations (critical and will differ by country)