Measuring Subjective Expectations in Surveys

1. Measuring Subjective Expectations in Surveys Adeline Delavande Institute for Social and Economic Research, University of Essex 2012 ESRC Research Methods Festival Survey Data Quality

2. Overview • Motivation • Measurement: how to elicit subjective expectations? • In developed countries • In developing countries • Data quality: Validation of answers • Basic patterns of answers • Violation of properties of probabilities • Correlation of expectations with characteristics • Design issues

3. Motivation • Social scientists and policy makers want to predict choice behavior • Most decisions are made under uncertainty • Deciding to retire • Choosing a medical treatment • Typical assumption: individuals maximize subjective expected utility • Basic identification problem with choice data alone: many combinations of expectations and preferences generate same choices

4. Motivation • One possibility to mitigate the identification problem • Elicit directly subjective expectations from survey respondents • Combine data on observed choice and subjective expectations to make inference on preferences

5. Motivation Other reasons to elicit subjective expectations from survey respondents • Find out whether there is systematic misperception in the population • Do women at risk of unintended pregnancy have misperception about effectiveness of birth control methods? • Ask expected choice behavior • Replace data on realizations • Subjective expectations about survival instead of long panel data on mortality to estimate differential mortality (Delavande and Rohwedder, forthcoming)

6. 2. Measurement: how to elicit subjective expectations?

7. Eliciting subjective expectations • Attitudinal research and verbal expectations data General Social Survey: Thinking about the next twelve months, how likely do you think it is that you will lose your job or be laid off—very likely, fairly likely, not too likely, or not at all likely? Survey of Consumers: Now turning to business conditions in the country as a whole—do you think that during the next 12 months we’ll have good times financially, or bad times, or what?

8. Eliciting subjective expectations • Attitudinal research and verbal expectations data: various problems • Interpretation of answers • Interpersonal comparability: Are the response comparable across people? • Very likely may mean different thing to different people • Intrapersonal comparability : Are the response comparable across event for the same individual? • Coarseness of response options • Researchers gain only limited information

9. Eliciting subjective expectations • Eliciting probabilistic expectations • Advantages: • well-defined numerical scale (comparability of answers) • Assessment of internal consistency of answers is possible • Do answers respect basic properties of probabilities? • Answers can be compared with known events frequencies

10. Eliciting subjective probabilities in developed countries Binary events • Typical wording: percent chance format Survey of Economic Expectations: What do you think is the percent chance that you will lose your job during the next 12 months? Health and Retirement Study: What do you think is the percent chance that you will live to be 75? My own survey: What do you think is the percentage chance that you would get pregnant during the next 12 months if you were using the birth control pill during that period?

11. Eliciting subjective probabilities in developed countries Distribution of beliefs about a continuous variable Two methods used so far: • Percent chance: elicit several points in the cumulative distribution On a scale from 0 to 100, where 0 means no chance and 100 means you are absolutely certain, what do you think is the percent chance that your Social Security benefits will be more than $1,000 per month? • Visual format in Internet surveys (Delavande and Rohwedder, 2008)

12. Eliciting distribution of beliefs about a continuous variable

13. Eliciting subjective probabilities in developing countries Binary events • Percent chance format (like in developed countries in a context of high literacy) • “Frequentist” (consider 20 people like you) • Visual format with physical objects (beans/stones)

14. Eliciting subjective probabilities in developing countries • Visual format (Delavande and Kohler, 2009) “I will ask you several questions about the chance or likelihood that certain events are going to happen. There are 10 beans in the cup. I would like you to choose some beans out of these 10 beans and put them in the plate to express what you think the likelihood or chance is of a specific event happening. One bean represents one chance out of 10.”

15. Eliciting subjective probabilities in developing countries Distribution of beliefs about a continuous variable • physical objects used to allocate probability mass into intervals (e.g., Delavande, Gine and McKenzie, 2011)

16. 3. Data quality: validations of expectations

17. Data quality: validations of expectations • Do people report their actual expectations?

18. Basic patterns of answers • Item non-response • Tend to be low: People willing and able to answer • Respondents use the whole scale from 0 to 100 • Some rounding to nearest 5 (see Manski and Molinari 2010) • Heterogeneity in answers • Often: focal answers at 0, 50, 100

19. Basic patterns of answers Probability of being alive at age 75 HRS 2006

20. Basic patterns of answers • Heaping at 0, 50 and 100 may proxy for respondents’ burden • 50-50: may reflect uncertainty rather than a true “1/2” probability • In HRS 2006, respondents who answer 50% to the subjective probability of survival are given a follow-up question asking whether they just do not know their survival probabilities or whether their belief is really that the chances are about 50 percent. It turns out that the fraction of 50s being simply uncertain is high – a little over 60 percent. • Rate of 50s tend to decrease with education and numeracy (Bruine de Bruin et al., 2000) • Rate of 50s varies with types of events

21. Violation of properties of probabilities • Do respondents understand concept of probabilities? • Delavande and Kohler (2008) ask about the probability of nested events in training questions in Malawi • Likelihood of going to the market within • 2 days • 2 weeks

22. Violation of properties of probabilities • 99% of respondents provide answers coherent with probability theory:  P(market within 2 days)<=P(market within 2 weeks)

23. Violation of properties of probabilities • Delavande, Gine and McKenzie (2011) ask a series of checks questions in a survey in India • Imagine I have 5 fishes, one of which is red and four of which are blue. If you pick one of these fishes without looking, how likely it is that you will pick the red fish? • Nested events • Probability zero event: How likely do you think it is that you will not catch any fish in the month of August if you go fishing 6 days a week? • Probability one event: How likely it is that you will eat fish at least once during the month of August?

24. Expectations and characteristics • Expectations vary with characteristics in the same way as actual outcomes

25. Likelihood of having to rely on family members in Malawi – by SES Source: Delavande and Kohler (2009)

26. Design issues • Which design for better quality?

27. Eliciting distribution of beliefs about a continuous variable • Comparing percent chance versus visual format (Delavande and Rohwedder, 2008) • Randomized design to ask about Social Security benefits expectations • HRS Internet survey in 2007 • Half sample was randomized into percent chance format (4 thresholds) • Half sample was randomized into visual format

28. Eliciting distribution of beliefs about a continuous variable

29. Eliciting distribution of beliefs about a continuous variable How do the 2 methods compare? • Non-response rate: less than 2% for both formats • Similar survey time • Respondents take only limited advantage of the additional precision of their format • Bins and balls: only 1 percent chose to use more than five bins • Percent chance: less than 3.5 percent of the answers per bin are not multiple of 5

30. Eliciting distribution of beliefs about a continuous variable • Unusable answers • Percent chance format: respondents do not respect the monotonicity across thresholds • 20% of respondents who answer the 4 thresholds • Those are less educated, less wealthy and in poorer health • Note: we did not inform respondents about the violation (different from Dominitz and Manski, 2006) • Visual format: respondents do not allocate all the balls • 3% of respondents

31. Eliciting distribution of beliefs about a continuous variable • Central tendency and spread of the distribution • The visual and percent chance format generate similar central tendency. • Respondents tend to allocate most probability mass in the middle bin • Yet, the percent chance format generates more spread-out distributions Average probability mass:

32. Eliciting distribution of beliefs about a continuous variable • Does the visual format create anchoring towards the middle of the visual aid? • Experiment in the RAND American Life Panel • Randomly allocate position of the bin containing the point estimate (middle bin or third bin) • Probability mass allocated in the bin containing the point estimate is not sensitive to the position of that bin

33. Eliciting distribution of beliefs about a continuous variable How to evaluate which format yields higher quality of answers? 1. Check how uncertainty in Social Security benefits correlates with uncertainty about related outcomes (subjective probability of reform, distance to claiming age, etc…) • Elicited uncertainty correlates with other sources of uncertainty in the expected direction for both formats • Both yield high-quality data from a substantive point of view

34. Eliciting distribution of beliefs about a continuous variable How to evaluate which format yield higher quality of answers? 2. Check whether uncertainty about future SS benefits predicts actual behavior related to retirement planning • Portfolio choice (Delavande and Rohwedder 2011) • Find that higher levels of uncertainty about future benefits hold a smaller share of their wealth in stocks in visual format only • Could also be due to smaller sample size or functional form assumptions that suit better visual format

35. Eliciting distribution of beliefs in developing countries • Delavande, Gine and McKenzie (2011) • Experiment in India to test the sensitivity of elicited expectations to variations in three facets of the elicitation methodology: • Number of beans (10 versus 20) • 20 beans: more precise answers but may increase respondent’s burden • Design of the support (pre-determined with 20 intervals or self-anchored with 4 intervals based on elicited min/max) • Self-anchor: more relevant to respondent but may increase interviewer’s burden • Ordering of questions

36. Eliciting distribution of beliefs in developing countries Subjective distribution elicited is robust to modifications in the elicitation design • When using a pre-determined support with many intervals, individuals with 20 beans use more intervals and have a distribution which is less uniform • 20 beans does not seem to unduly increase respondent burden and allows more precision • Compare individual-level realized distribution of daily catches during August to elicited distribution • Greatest accuracy occurs when using 20 beans and the pre-determined support, which results in a 38 percent improvement in accuracy relative to using 10 beans and a self-anchored support

37. Conclusions • Various ways can be used to assess expectations data quality • Introduce training questions to check whether respondents understand the concept of probabilities • Correlate with characteristics • Evaluate whether expectations predict actual behavior • We may be able to improve data quality with more innovative elicitation formats

38. References • Bruine de Bruin, W¨andi, Baruch Fischhoff, Susan G. Millstein, and Bonnie L. Halpern-Felsher (2000). “Verbal and Numerical Expressions of Probability: ‘It’s a Fifty-Fifty Chance’.” Organizational Behavior and Human Decision Processes 81:115–31. • Delavande, Adeline, Xavier Giné and David McKenzie (2011) “Eliciting Probabilistic Expectations with Visual Aids in Developing Countries: How sensitive are answers to variations in elicitation design?”, Journal of Applied Econometrics. • Delavande, Adeline and Hans-Peter Kohler (2009) “Subjective Expectations in the Context of HIV/AIDS in Malawi”, Demographic Research. • Delavande, Adeline and Susann Rohwedder (2008). “Eliciting Subjective Expectations in Internet Surveys,” Public Opinion Quarterly. • Delavande, Adeline and Susann Rohwedder (2011). “Individuals’ Uncertainty about Future Social Security Benefits and Portfolio Choice ,” Journal of Applied Econometrics. • Delavande, Adeline and Susann Rohwedder (forthcoming). “Differential survival in Europe and the US: Estimates based on subjective probabilities of survival ,” Demography. • Manski, Charles F. and Francesca Molinari (2010). “Rounding Probabilistic Expectations in Surveys,” Journal of Business & Economic Statistics.