Survey Response Rates: Trends and Standards

1. Survey Response Rates: Trends and Standards Karen Donelan, ScD Senior Scientist in Health Policy Massachusetts General Hospital/Harvard Medical School AcademyHealth - June 27th, 2006

2. Overview • What is a response rate? • Why do we care about response rates? • Trends • Does nonresponse = biased response? • How should rates be calculated and reported? • How can we improve?

3. What is a Response Rate? • The number of complete interviews with reporting units divided by the number of eligible reporting units in the sample

4. Other Rates • Cooperation rates - The proportion of all cases interviewed of all eligible units ever contacted. • Refusal rates - The proportion of all cases in which a sampled unit or the respondent refuses to be interviewed, or breaks-off an interview, of all potentially eligible cases. • Contact rates - The proportion of all cases in which some responsible sampled unit member was reached.

5. Why Do We Care About Response Rates? • Sources of Error in Surveys • Coverage, Measurement, Nonresponse, Processing, Sampling • Practical implications • Generalizability • Publishability (often cutoff or major test of quality) • Credibility • Fundability

6. What Do We Know About Response Rate Trends?

7. Current Population Survey (CPS) http://www.census.gov/prod/2000pubs/tp63.pdf

8. Behavioral Risk Factors Survey (BRFSS) : Telephone Survey Source: Groves et al. 2004

9. U Michigan: Survey of Consumers Source: Groves et al. 2004

10. Why Does Nonresponse Happen? • Who is in the sample? (demographics, SES, lifestyle) • # Contacts • Schedule of contacts • Mode of contact • Respondent selection • Respondent cooperation (refusal) • Incentives ($, benefit) • Respondent burden (time, boredom, frustration) • Salience • Respondent ability to respond to questions • Sponsorship • Privacy concerns

11. How Should Rates Be Calculated and Reported? Disclosure • Units drawn and attempted • Addresses mailed • Phone numbers called • Households approached • Patients invited • Sample Disposition: what happened? • Eligible and ineligible respondents • Bad contact information • How many attempts, at what intervals • # Never reached, # actually contacted • # completed, refused, never got a decision

12. www.aapor.org

13. Does Nonresponse = Biased Response? • Low response rates may not be problematic • Representativeness doesn’t necessarily increase with response rate • Sample representativeness is a function of the difference between respondents and nonrespondents on the statistic(s) of interest • High response rates can can yield an unrepresentative sample (high nonresponse bias) • Low response rates can still yield a representative sample • Keeter et al. (2000), Curtin, Presser and Singer (2000), Merkle and Edelman (2002)

14. Measuring Nonresponse Bias • With response rates falling, understanding the impact on study findings is essential • Assess quantitatively if possible, but at least consider qualitatively • Methods for assessing nonresponse bias: • Response rate comparisons across groups • Follow-up interviews with non-respondents • Comparing early vs. late respondents • Comparisons to similar estimates from other sources • Post survey adjustment for nonresponse

15. Ways Not to Approach Nonresponse • Ignore it • Obscure it • Omit it • Fail to measure/comment on potential associated bias • Give up on it

16. Improving Response Rates • Multiple contacts • Multiple modes of contact • Interviewer training • Advance Notification/Endorsement • Incentives • Reduce respondent burden • Increase relevance to respondent • Time and money to get quality

17. Reporting Response Rates • Know and apply standards (AAPOR, CASRO) • Engage and manage with vendors • Structure sample disposition • Response and nonresponse analysis • Publish enough information to allow others to do the calculations • Peer review: build awareness of standards