Nonresponse Rates and Nonresponse Bias In Surveys

1. Nonresponse Rates and Nonresponse Bias In Surveys Robert M. Groves University of Michigan and Joint Program in Survey Methodology, USA Emilia Peytcheva University of Michigan, USA Funding from the Methodology, Measurement, and Statistics Program of the US National Science Foundation, Grant 0297435

2. Four Mutually-Problematic Observations • With 100% response rates probability sampling offers an inferential paradigm with measurable uncertainties for unbiased estimates • Response rates are declining • Keeter et al. (2000), Curtin et al. (2000), Merkle and Edelman (2002) show no nonresponse bias associated with varying nonresponse rates • Practitioners are urged to achieve high response rates Result: Confusion among practitioners

3. Assembly of Prior Studies of Nonresponse Bias • Search of peer-reviewed and other publications • 47 articles reporting 59 studies • About 959 separate estimates (566 percentages) • mean nonresponse rate is 36% • mean bias is 8% of the full sample estimate • We treat this as 959 observations, weighted by sample sizes, multiply-imputed for item missing data, standard errors reflecting clustering into 59 studies and imputation variance

4. Percentage Absolute Relative Bias

5. Percentage Absolute Relative Nonresponse Bias by Nonresponse Rate for 959 Estimates from 59 Studies

6. Conclusions from 959 Estimates • Examples of large nonresponse bias exist • Variation in nonresponse bias lies mostly among estimates within the same survey • The nonresponse rate by itself is not a good predictor of nonresponse bias • [Note: We cannot infer from the scatterplot about what would happen within a study if response rates were increased]

7. Thinking Causally About Nonresponse Rates and Nonresponse Error • Key scientific question concerns mechanisms of response propensity that create covariance with survey variable where is the covariance between the survey variable, y, and the response propensity, p • What mechanisms produce the covariance?

8. Alternative Causal Models for Studies of Nonresponse Rates and Nonresponse Bias

9. Types of Hypotheses about Influences on Nonresponse Error • Influences on response rates • urbanicity, gender, age • topic of survey, population’s interest in topic • mode of data collection • prenotification, incentives • Sponsorship • prior involvement of population with sponsor • government vs. other sponsor • Types of measures • attitudinal vs. behavioral • questions related to topic of survey vs. others • Type of statistic • means on counts/continuous variables vs. percentages • differences of subclass means

10. Types of Hypotheses about Influences on Nonresponse Error • Influences on response rates • urbanicity, gender, age • topic of survey, population’s interest in topic • mode of data collection • prenotification, incentives • Sponsorship • prior involvement of population with sponsor • government vs. other sponsor • Types of measures • attitudinal vs. behavioral • questions related to topic of survey vs. others • Type of statistic • mean vs. percentages • differences of subclass means Attributes of Surveys Attributes of Estimates

11. Exploratory Analysis in Two Steps Examine only estimates that are percentages, using standardized values of the percentages • Step 1: pooling all 566 estimates, examine , difference of respondent and nonrespondent means • Step 2: separating the estimates by their survey’s response rate, examine , nonresponse bias

12. Respondent-Nonrespondent Functions on Standardized Percentage Estimates by Type of Population diff=.075 ste=.033

13. Respondent-Nonrespondent Functions on Standardized Percentage Estimates by Type of Population diff=.075 diff= .013 .0013 .045 ste=.033 ste= .0092 .0058 .020

14. Respondent-Nonrespondent Functions on Standardized Percentage Estimates by Mode diff=.040 diff= .00091 .0012 .025 ste=.024 ste= .0011 .0055 .011

15. Respondent-Nonrespondent Functions on Standardized Percentage Estimates by Involvement with Sponsor diff=.052 diff= .017 .0014 .029 ste=.022 ste= .0079 .0063 .010

16. Respondent-Nonrespondent Functions on Standardized Percentage Estimates by Type of Measure diff=.14 diff= .084 ---- .070 ste=.022 ste= .0074 ---- .016

17. Respondent-Nonrespondent Functions on Standardized Percentage Estimates by Statistic’s Relevance to Topic diff=.006 diff= .000 -.0014 .014 ste=.023 ste= .0074 .0091 .025

18. Respondent-Nonrespondent Functions on All Estimates by Type of Estimator diff=4.37 diff= 1.07 1.38 2.14 ste=0.72 ste= 0.38 0.39 0.51

19. Do Differences of Subclass Means have Lower Nonresponse Bias? • When estimating subclass differences, we hope that nonresponse biases of the two estimates cancel • 120 reported estimates of subclass means and their differences • Only 45 of them have bias of the differences of subclass means lower than average bias of the two subclass means • this comports with only 45 having two subclass means with biases of the same sign

20. Absolute Value of Bias of Difference of Subclass Mean by Absolute Value of Subclass Mean

21. Types of Hypotheses about Influences on Nonresponse Error • Influences on response rates • urbanicity, gender, age • topic of survey, population’s interest in topic • mode of data collection • prenotification, incentives • Sponsorship • prior involvement of population with sponsor • government vs. other sponsor • Types of measures • attitudinal vs. behavioral • questions related to topic of survey vs. others • Type of statistic • mean vs. percentages • differences of subclass means

22. Five Summary Statements • Large nonresponse biases exist • Most variation in nonresponse biases lie among estimates in the same survey • Some types of surveys are more susceptible to biases (e.g., interviewer-administered, studies of population without prior involvement with the sponsor, general population surveys) • Some types of estimates seem more susceptible to biases (e.g., measures of attitudes, percentages, (maybe) estimates related to survey topic) • Differences of subclass means do not tend to have lower biases that the individual subclass means Note: preliminary and exploratory analysis; there is much more to do