Protocol & preliminary results from a transnational cognitive interviewing study: Ensuring valid cross-cultural comp

1. TTURC Planning Meeting, Portland, OR, USA March 2, 2008 Protocol & preliminary results from a transnational cognitive interviewing study: Ensuring valid cross-cultural comparisons James F. Thrasher, Anne C. K. Quah, Ron Borland, Rahmat Awang, Maizurah Omar, Buppha Sirirassamee, Marcelo Boado, Ashlee Watts, Ana Dorantes Alonso INSTITUTO NACIONAL DE SALUD PÚBLICA MÉXICO

2. Purpose of the study • Develop a protocol for assessing and reducing systematic response error in multi-lingual and multi-cultural surveys • Focus on existing ITC survey questions: • Assess equivalence of comprehension and response error across 6 countries, 4 languages; • Recommend alterations to questions to increase equivalence in later studies. • Increase data quality  more confidence in results from comparative data analyses 2

3. Questionnaire protocol • Ask participant survey question. • Passive recording of behaviors that indicate comprehension problems (i.e., behavioral coding*). • Ask follow-up questions that aim to assess response error across the steps involved in cognitive processing of survey questions (i.e., cognitive interviewing). *Johnson, TP et al (2006). Cultural variability in the comprehension of health survey questions. Annals of Epidemiology, 16, 661-668. 3

4. Study Steps Identify candidate questions Determine likely biases Develop probes to assess biases Develop translator guide Pilot protocol Review pilot data Translate protocol Train interviewers Translate responses to English Adjust protocol Implement protocol Enter data Identify themes in qualitative data Determine evidence for systematic bias Review quantitative data Suggest changes 4

5. Sample Characteristics* *n=20 for each country 5

6. Nicotine in cigarettes is the chemical that causes most of the cancer: False 6

7. Behavioral codes indicate some systematic bias 7

8. Can you tell me what you know about nicotine? 8

9. Conclusions & Recommendation • Behavioral codes  systematic bias • Awkward wording? • Acquiescence bias? • “don’t know” or haven’t heard of nicotine • Order effect after health outcome questions? Recommendation: Clarify question intent: • Assess knowledge? Why does this matter? Could first ask if they have heard of nicotine • ITC-SEA W3: Cigarettes contain nicotine 9

10. Tobacco is addictive: “agree” or “strongly agree” 10

11. Behavioral Codes indicate no problem with “Tobacco is addictive” 11

12. Other things that are addictive 12

13. What does it mean to say that something is addictive? It's got a grip of you and you can't let go. It's a must have, you've gotta have it. 39 year old Australian male The body gets tolerance for it and the body won't function normally with out it. 27 year old US female If he doesn’t get that thing, he cannot do other things…His mind is always imagining that thing…if he doesn’t get it, he cannot sit still. 26 year old Malaysian male 13

14. Conclusions & Recommendations • Behavioral coding  no clear bias • Connotations of and associations with addiction appear to differ across countries Recommendation: • Ask question(s) that focus specifically on: • Control over smoking (e.g., Smokers cannot go long without smoking, even when they don’t want to smoke) • Control over quitting (e.g., Completely stopping smoking is extremely difficult) 14

15. Lights are less harmful than regular cigarettes: “Disagree”/”Strongly disagree” 15

16. Behavioral codes indicate no systematic bias 16

17. Other differences beween light and regular cigarettes? 17

18. Conclusions & Recommendations • Behavioral coding  no systematic bias • Perceptions of light vs. regular appear to differ across countries • Focus on product characteristics in western countries • Focus on sensory effects of product use in non-western countries Recommendation • Be aware of the possibility of the salience of these characteristics when developing new questions about product risks and effects. 18

19. Noticed warning labels “often” or “very often” in last month 19

20. Behavioral codes indicate some evidence of systematic bias 20

21. Follow up probes indicate incorrect reference period and issues with the term “notice” 21

22. Conclusions & Recommendations • Behavioral coding  indicates systematic bias (*problems with US & Australia) • Consistently incorrect identification of reference period for the question • Some indications of systematic differences in meanings of “notice” Recommendation: • Ask about last time when “noticed” • Review terms used for translation of “notice” 22

23. In last 6 months, noticed ads on… 23

24. Conclusions & Recommendations • Behavioral coding  no systematic bias • Across channels, participants in all countries found it difficult to remember whether they had seen an ad • Some indication of systematic bias in difficulty of remembering exposure for TV and radio. • Consistently incorrect identification of reference period for noticing Recommendation: • Ask about last time “saw”, with options for “never” and for “more than 6 months ago” 24

25. General Conclusions • Cognitive interviewing and behavioral coding techniques can be implemented across linguistic and cultural settings • Overcoming challenges of assessing equivalence across languages demands collaboration from multilingual partners. • Comprehension of some questions appeared generally comparable whereas other questions appear to generate response error and need to be adjusted. 25

26. General Recommendations • Any changes to harmonize item comprehension should be verified through another round of cognitive interviews. • Interviewers should be better trained to clarify ambiguous or repetitive responses. 26

27. General Recommendations • Prepare translation notes (i.e., definitions of the concept being measured & question intent) for new questions. • Choose source language questions that use simple, translatable language • “Notice” vs. “saw” • To avoid augmenting “translation loss”, questions should originate from one source. • NOT: English  Spanish  Malay • Equivalence of comprehension may demand flexibility in “source” language item 27

28. General Recommendations Systematic documentation of the evolution, translation, and meanings of ITC survey questions, key terms, and phrasing across countries. • Translator’s notes • May include possible translation equivalents in the target languages • Provides record of original survey item wording • Allows future researchers to understand intentions behind survey items and any changes to them 28

29. International Tobacco Control Policy Evaluation Projecthttp://www.itcproject.org http://www.roswelltturc.org ITC Project Research Support Core support provided by the U.S. National Cancer Institute to the Roswell Park TTURC (P50 CA111236) Additional major funding provided by the Canadian Institutes of Health Research

30. Study Steps Identify candidate questions Determine likely biases Develop probes to assess biases Develop translator guide Pilot protocol Review pilot data Translate protocol Train interviewers Translate responses to English Adjust protocol Implement protocol Enter data Identify themes in qualitative data Determine evidence for systematic bias Review quantitative data Suggest changes 30