How Much of Interviewer Variance is Really Nonresponse Error Variance ?

1. How Much of Interviewer Varianceis Really Nonresponse Error Variance? Brady T. West Michigan Program in Survey Methodology University of Michigan-Ann Arbor Kristen Olson Survey Research and Methodology Program University of Nebraska-Lincoln June 14, 2010 International Total Survey Error Workshop 2010 1

2. Interviewer Variance: The Problem • An undesirable product of the data collection process, given interpenetrated sample designs • Responses for same interviewer are more similar than responses for different interviewers • Leads to inflation of variance in survey estimates due to intra-interviewer correlation, ρint • ρint = 0.01, 30 cases per interviewer  13.6% increase in SE of estimates • ρint usually less than 0.02, but can be larger 2

3. Research Question • Does ρint arise from complex interviewer-respondent interactions / probing for hard items? • High estimates of ρint (0.03–0.12) for factual (easy) and self-completion items in literature… • There is also consistent empirical evidence of interviewer variance in response rates • One estimates ρint with respondent data only, ignoring contributions of NR error variance • How much of interviewer variance can be attributed to nonresponse error variance? 3

4. The Wisconsin Divorce Study (WDS) • SRS of divorce records from four Wisconsin counties in 1989 and 1993 • Sampled divorce records included official information also collected in a survey • The present study focuses on data collected using CATI: interviewer effects are likely attenuated relative to CAPI • n = 733 cases randomly sampled, and 355 CATI interviews performed by 31 trained interviewers 4

5. WDS Data • Six Survey Variables of Interest • Length of Marriage in Months • Time since Divorce in Months • Time since Marriage in Months • Number of Marriages including the Divorce • Age at Marriage • Age at Divorce • Date of Divorce was recorded by an official body; other frame measures were reported by one member of the couple (possible errors) 5

6. Assigning Nonrespondents • Ideally, cases would not be worked by multiple interviewers (e.g., Singer and Frankel, 1982) • WDS used refusal conversion, and there were frequent changes in interviewers working non-finalized cases • This complicates the process of assigning nonrespondents to interviewers 6

7. Assigning Nonrespondents • Assumption: Interviewers working a particular shift work a random subsample of cases • The focus of this study is on interviewer variance within a shift, rather than across shifts • Persons with different characteristics are likely to be contacted at different times of the day • Account for shift: avoids possible confounding of nonresponse error with differences across shifts 7

8. Assigning Nonrespondents • Definitions of shifts: • Weekday, 9-5pm (Shift 1: 26.8% of calls) • Weekday, after 5pm (Shift 2: 44.1% of calls) • Weekend, any time (Shift 3: 29.1% of calls) • Similar to work of Stokes and Yeh (1988) • Interviewers worked multiple shifts • Alternative shifts were also considered, and the study results did not change 8

9. Assigning Nonrespondents • Respondents were assigned to the interviewer completing the interview • Contacted refusals were assigned to: • the first interviewer receiving a refusal, or • the last interviewer to make contact • Non-contacts were assigned to the last interviewer making a call to the case • Random assignment of non-contacts was also considered; no change in results 9

10. Assigning Nonrespondents • Limited power: 19-24 interviewers worked each shift, based on assignments • Large variability in assigned workloads across interviewers within a shift • Between 8 and 15 cases per interviewer within a shift, on average • Response rates lowest during the week, and especially before 5pm (41.1%) 10

11. Analytic Approach • Examine interpenetration assumptions • Estimate ρint for each survey variable within each shift, based on respondent data • Test interviewer variance for significance • Estimate all variance components of the MSE of the respondent mean (possible with WDS data) • Estimate interviewer effects on (and interviewer contributions to) the variance components • Compute the proportion of interviewer-contributed variance due to NR error variance 11

12. Example Derivation (Groves and Magilavy, 1984) • MSE of respondent mean (u = # of refusals): • Example: Refusal error component 12

13. Example Derivation, cont’d • Estimate variance components using linearized variance estimators • Accounts for clustering due to interviewers and unequal workloads • Estimate interviewer effects on variances based on estimates of intra-interviewer correlations in true values (Census Bureau, 1985) 13

14. Example Derivation, cont’d • Estimated total contribution of interviewers to refusal error variance: • The estimated contribution is a function of intra-interviewer correlations in true values, for respondents and refusals • Similar derivations for other components 14

15. Results: Interviewer Variance Based on Respondent Data • Interpenetration evident in each shift • Four variable / shift pairs were found to have unusually large estimates of ρint: • Age at Divorce, Shift 2 (ρint = 0.08, p = 0.05) • Age at Divorce, Shift 3 (ρint = 0.10, p = 0.11) • Age at Marriage, Shift 2 (ρint = 0.11, p = 0.01) • Mths. since Marr., Shift 2 (ρint = 0.05, p = 0.13) 15

16. Results: Sources of Interviewer Variance in Age at Divorce (Shift 2) • Estimated intra-interviewer correlations in • Response errors: -0.003 • True values for respondents: 0.092 • True values for refusals: 0.008 • True values for noncontacts: -0.055 • Total estimated variance of R mean: 0.423 • Total estimated variance contributed by interviewers: 0.033 • Additional variance contributed by interviewers is due to intra-interviewer correlations in true values for respondents! 16

17. Additional Results • Similar findings for age at divorce in shift 3 • Response error variance was main contributor for age at marriage in shift 2 and months since marriage in shift 2 • Response error variance may arise from outliers, as shown in the following graph

18. Illustration of Variance Sources

19. Conclusions • Interviewer variance on key survey variables may arise from nonresponse error variance among interviewers • Interviewers may successfully obtain cooperation from different pools of respondents (e.g., older vs. younger) • Liking theory could be one explanation: variance in interviewer ages, voices  variance in respondent ages (F. Conrad) 19

20. Implications for Practice / Future Work • Monitoring Strategies: managers can continuously compare available features of R and NR for each interviewer, and intervene when large differences arise • Findings need to be replicated in a face-to-face setting with interpenetrated subsamples assigned to interviewers • Access to interviewer features would also enable use of multilevel modeling 20

21. Thank You! • Bob Groves, Mick Couper, Frauke Kreuter and Paul Biemer have provided very helpful feedback and comments • Thanks to Vaughn Call for providing access to the WDS data • Please email bwest@umich.edu for these slides or a draft of the paper 21