Optimal Adaptive Survey Design

1. Optimal Adaptive Survey Design Lars Lyberg, Frauke Kreuter, and James Wagner ITSEW 2010 Stowe, VT, USA, June 16

3. What Should Be Designed? • Requirements+specifications+operations • Ideal goal+ Defined goal+Actual results • Good survey design means control of accuracy through the specs (QA) and control of operations (QC)

4. Some Early Thinking • Hansen-Hurwitz-Pritzker 1967 • Take all error sources into account • Minimize all biases and select a minimum-variance scheme so that Var becomes an approximation of (a decent) MSE • The zero defects movement that later became Six Sigma • Dalenius 1969 • Total survey design

5. Some More Thinking • Textbook on total survey design • Hansen-Hurwitz-Cochran-Dalenius • Survey models and specific error sources • Cochran’s comment from 1968

6. Alternative Criteria of Effectiveness • Minimizing MSE for a given budget while meeting other requirements • Maximizing fitness for use for a given budget • Maximizing comparability for a given budget • All these reversed • Something else?

7. The Elements of Design • Assessing the survey situation (requirements) • Choosing methods, procedures, “intensities”, and controls (specifications) • Allocating resources • Assessing alternative designs • Carry out one of them or a modification of it • Have a Plan B

8. So, What’s the Problem? • No established survey planning theory • Multi-purpose, many users • The information paradox • Uninformed clients/users/designers • Much design work is partial, not total • Limited knowledge of effects of measures on MSE and cost

9. More Problems • Decision theory and economics theory not used to their potential • New surveys conducted without sufficient consideration of what is already known • No one knows the proper allocation of resources put in before, during and after • The literature is small

10. Various Skills Needed Which Calls for a Design Team • Survey methodology • Subject-matter • Statistics (decision theory, risk analysis, loss functions, optimization, process control) • Economics (cost functions, utility) • IT

11. The Adaptive Element • The entire survey process should be responsive to anticipated uncertainties that exist before the process begins and to real time information obtained throughout the execution of the process or • Use process data (paradata) to check, and if necessary, adjust the process

12. We Should Assemble What We Know • Assessment methods • Design principles • Trade-offs and their effects • The potential offered by other disciplines • We shouldn’t accept partial designs

13. Apply Design Principles • If pop is skewed then…. • If pop is nested then…. • If questions are sensitive then…. • If a high NR rate is expected then…

14. Apply SOPs, CBMs or Best Practices • Part of the design is to use known, dependable methods

15. Examples of Trade-offs • Accuracy vs timeliness • Response burden vs wealth of detail • Conduct survey vs other information collection • Large n vs smaller n • Mixed vs single mode • NR bias vs measurement error • NR vs interpretation by family members

16. Process view • Upstream thinking (prevention) • Understanding variation • Measure cost of poor quality and waste • Intervention or improvement actions should be based on good data and statistical analysis • Continuous monitoring

17. Tentative Course Syllabus • The elements of design • Real world examples (e.g., CPS Technical Paper 63, PIAAC, the Monthly Retail Trade Survey, the Annual Survey of Hale Mountain Fish & Game Club, VT) • The literature on optimal decisions • Theory for adaptive treatment design and risk management

18. Course syllabus continued • Data for monitoring and decision making • Analysis of such data • Design lessons learned • Examples of bad designs and not so great trade-offs • Student project with TSE perspective • Student presentations