Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting Measures November 13, 2008

1. Class 9 Interpreting Pretest Data, Considerations in Modifying or Adapting MeasuresNovember 13, 2008 Anita L. Stewart Institute for Health & Aging University of California, San Francisco

2. Overview of Class 9 • Analyzing pretest data • Modifying/adapting measures • Keeping track of your study measures • Creating and testing scales in your sample

3. Summarize Data on Pretest Interviews • Summarize problems and nature of problems for each item • Determine how important problems are • Results become basis for possible revisions/adaptations

4. Methods of Analysis • Optimal: transcripts of all pretest interviews • For each item - summarize all problems • Analyze dialogue (narrative) for clues to solve problems

5. Behavioral Coding • Systematic approach to identifying problems with items • “interviewer” and “respondent” problems • Can code problems based on: • Standard administration • Responses to specific probes

6. Examples of Interviewer “Behaviors” Indicating Problem Items • Question misread or altered • Slight change – meaning not affected • Major change – alters meaning • Question skipped

7. Examples of Respondent “Behaviors” Indicating Problem Items • Asks for clarification or repeat of question • Did not understand question • Doesn’t know the answer • Qualified answer (e.g., it depends) • Indicates answer falls between existing response choices • Refusal

8. Summarize Behavioral Coding For Each Item • Proportion of interviews (respondents) with each problematic behavior • # of occurrences of problem divided by N • 7/48 respondents requested clarification

9. Behavioral Coding Summary Sheet: Standard Administration (N=20)

10. Can Identify Problems Even When No Problem “Behaviors” Found • Respondents appear to answer question appropriately • Additional problems identified with probes • Probe on meaning: Response indicates lack of understanding • Probe on use of response options: Response indicates options are problematic

11. Behavioral Coding of Probe Results I asked you how often doctors asked you about your health beliefs. What does the term “health beliefs” mean to you? Behavioral coding: # times response indicated lack of understanding as intended • e.g., 2/15 respondents did not understand meaning based on response to probe

12. Behavioral Coding Summary: Standard Administration (N=20) + Probes (N=10)

13. Interpret Behavioral Coding Results • Determine if problems are common • Items with only a few problems may be fine • Quantifying “common” problems • several types of problems (many row entries) • several subjects experienced a problem • problem w/item identified in >15% of interviews

14. Continue Analyzing Items with “Common” Problems • Identify “serious” common problems • Gross misunderstanding of the question • Yields completely erroneous answer • Couldn’t answer the question at all • Some less serious problems can be addressed by improved instructions or a slight modification

15. Addressing More Serious Problems • Conduct content analysis of transcript • Use qualitative analysis software (e.g., NVIVO) • For these items: review dialogue that ensued during administration of item and probes • can reveal source of problems • can help in deciding whether to keep, modify or drop items

16. Results: Probing Meaning of Phrase • I asked you how often doctors asked you about your health beliefs? What does the term ‘health beliefs’ mean to you? “.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in going to doctors?”

17. Results: Probing Meaning of a Phrase • What does the phrase “office staff” mean to you? “the receptionist and the nurses” “nurses and appointment people” “the person who takes your blood pressure and the clerk in the front office”

18. Results: Probing Meaning of Phrase • On about how many of the past 7 days did you eat foods that are high in fiber, like whole grains, raw fruits, and raw vegetables? • Probe: what does the term “high fiber” mean to you? • Behavioral coding of item • Over half of respondents exhibited a problem • Review answers to probe • Over ¼ did not understand the term Blixt S et al., Proceedings of section on survey research methods,American Statistical Association, 1993:1442.

19. Results: No Behavior Coding Issues but Probe Detected Problems • I seem to get sick a little easier than other people (definitely true, mostly true, mostly false, definitely false) • Behavioral coding of item • Very few problems • Review answers to probe • Almost 3/4 had comprehension problems • Most problems around term “mostly” (either its true or its not) Blixt S et al., Proceedings of section on survey research methods,American Statistical Association, 1993:1442.

20. Results: Beck Depression Inventory (BDI) and Literacy • Cognitive interviews: older adults, oncology pts, and less educated adults • Administered REALM (reading literacy test) and some selected BDI items • Asked to paraphrase items TL Sentell, Community Mental Health Journal, 2008;39:323

21. Results: Beck Depression Inventory (BDI) and Literacy (cont) • For each item, from 0-62% correctly paraphrased item • Most misunderstandings: vocabulary confusion • Phrase: I am critical of myself for my weaknesses and mistakes • “Critical is when you’re very sick” • “I don’t know how to explain mistakes”

22. Interpreting Pretest Results of Self-Administered Questionnaires • Missing data is a clue to problematic items • More missing data associated with unclear, difficult, or irrelevant items • Cognitive interviewing can help determine reasons for missing data

23. How Missing Data Prevalence Helps • Items with large percent of responses missing – clue to problem • In H-CAHPS® pretest: Did hospital staff talk with you about whether you would have the help you needed when you left the hospital? • 35% missing for Spanish group • 29% missing for English group MP Hurtado et al. Health Serv Res, 2005;40-6, Part II:2140-2161

24. Exploring Differences by Diverse Groups • Back to issue of “equivalence” of meaning across groups • All cognitive interview analyses can be done separately by group

25. Results: Use of Response Scale • Do diverse groups use the response scale in similar ways? • Re questions about cultural competence of providers • Interviewers reported that Asian respondents who were completely satisfied did not like to use the highest score on the rating scale California Pan-Ethnic Health Network (CPEHN) Report, 2001

26. Results: Use of Response Scale (cont) • Behavioral Risk Factor Surveillance Survey (BRFSS) pretesting • Found that Puerto Rican, Mexican American, and African American respondents more likely to choose extreme response categories than Whites. RB Warnecke et al, Ann Epidemiol, 1997:7:334-342

27. Differential Use of CAHPS® 0-10 Global Rating Scale • Compared Medicaid and commercially insured adults on use of scale • Medicaid enrollees more likely than commercial participants to use extreme ends of scale • All other things being equal PC Damiano et al, Health Serv Outcomes Res Method, 2004:5:193-205

28. Results: Probe on Difficulty:CES-D Item “During the past week, how often have you felt that you could not shake off the blues, even with help from family and friends” • Probe: Do you feel this is a question that people would or would not have difficulty understanding? • Latinos more likely than other groups to report people would have difficulty TP Johnson, Health Survey Research Methods, 1996

29. Overview of Class 9 • Analyzing pretest data • Modifying/adapting measures • Keeping track of your study measures • Creating and testing scales in your sample

30. Now What! • Issues in adapting measures based on pretest results • Cognitive interview pretesting during development phases of measure • Can modify items and continue pretesting • Cognitive interview pretesting prior to using published survey: • More problematic

31. Modification: Probing the Meaning of a Phrase • What does the phrase “office staff” mean to you? “the receptionist and the nurses” “nurses and appointment people” “the person who takes your blood pressure and the clerk in the front office” • We changed the question to receptionist and appointment staff

32. Results: Probing Meaning and Cultural Appropriateness • I asked you how often doctors asked you about your health beliefs? What does the term ‘health beliefs’ mean to you? “.. I don’t want medicine” “.. How I feel, if I was exercising…” “.. Like religion? --not believing in going to doctors?” • We changed the question to “personal beliefs about your health

33. Criteria for Whether or Not to Modify Measure • Contact author • May be open to modifications, working with you • Be sure your opinion is based on extensive pretests with consistent problems • Don’t rely on a few comments in a small pretest • Work with a measurement specialist to assure that proposed modifications are likely to solve problem

34. Tradeoffs of Using Adapted Measures Advantages • Improve internal validity Disadvantages • Lose external validity • Know less about modified measure • Need to defend new measure

35. Adding New (Modified) Items • One approach if you find serious problems with a standard measure • Write new items you think will be better (use same format) • Retain original intact items and add modified items • Can test original scale and revised scale with modified items instead of original items

36. Modifying response categories • If response choices are too few and/or coarse, can improve without compromising too much • Try adding levels within existing response scale

37. SF36 version 1 1 - All of the time 2 - Most of the time 3 - A good bit of the time 4 - Some of the time 5 - A little of the time 6 - None of the time SF36 version 2 1 - All of the time 2 - Most of the time 3 - Some of the time 4 - A little of the time 5 - None of the time One Modification: Too Many Response Choices

38. Usual responses: 1 - Definitely true 2 - Mostly true 3 - Don’t know 4 - Mostly false 5 - Definitely false Modified: 1 – Not at all true 2 – A little true 3 - Somewhat true 4 - Mostly true 5 – Definitely true Modification of Health Perceptions Response Choices for Thai Translation e.g., My health is excellent, I expect my health to get worse

39. Modifying Item Stems • If item wording will not be clear to your population • Can add parenthetical phrases • Have you ever been told by a doctor that you have diabetes (high blood sugar)?

40. Strategy for Modified Measures • Test measure in original and adapted form • Choose measure that performs the best

41. Analyzing New (Modified) Measure • Factor analysis • All original items • Original plus new items replacing original • Correlations with other variables • Does the new measure detect stronger associations? • Outcome measure • Does the new measure detect more change over time?

42. Usual classifications 0-9, 10 (dichotomy) Proposed classification 0-8, 9-10 Analytic Strategy: CAHPS® 0-10 Global Rating Scale: Response Can’t change the scale – part of standardized survey PC Damiano et al, Health Serv Outcomes Res Method, 2004:5:193-205

43. Overview of Class 9 • Analyzing pretest data • Modifying/adapting measures • Keeping track of your study measures • Creating and testing scales in your sample

44. Questionnaire Guides • Organizing your survey measures • Keep track of measurement decisions • Sample guide to measures (last week) • Documents sources of measures • Any modifications, reason for modification

45. “Sample Guide to Measures” Handout • Type of variable • Concept • Measure • Data source • Number of items/survey question numbers • Number of scores or scales for each measure • References

46. Sample “Summary of Survey Variables..” Handout • Develop “codebook” of scoring rules • Several purposes • Variable list • Meaning of scores (direction of high score) • Special coding • How missing data handled • Type of variable (helps in analyses)

47. Item Naming Conventions • Optimal coding is to assign raw items their questionnaire number • Can always link back to questionnaire easily • Some people assign a variable name to the questionnaire item • This will drive you crazy

48. Variable Naming Conventions • Assigning variable names is an important step • make them as meaningful as possible • plan them for all questionnaires at the beginning • For study with more than one source of data, a suffix can indicate which point in time and which questionnaire • B for baseline, 6 for 6-month, Y for one year • M for medical history, L for lab tests

49. Variable Naming Conventions (cont) Medical History Questionnaire HYPERTMB HYPERTM6 Baseline 6 months

50. Variable Naming Conventions (cont) • A prefix can help sort variable groupings alphabetically • e.g., S for symptoms SPAINB, SFATIGB, SSOBB