The Philosophy of French experiments on Internet and mixed modes data collection

1. Tiaray Razafindranovona September 26th, 2013 The Philosophy of French experimentson Internet and mixed modes data collection

2. Introduction • Better quality, more complexity … • … in a general context of budgetary reduction ! • Mixed modes to face these challenges ? • Experimental surveys will help to answer

3. Principles of French experiments • An experiment for (almost) each new household survey • General results on mixed modes surveys not sufficient • Experiments in parallel of each new CAPI survey • No interference with main survey • Workload • Sampling frames • Each survey must test something new • Accumulation of experience • Test of new specific points not already handled

4. Standard protocol • Sample frames • Name and adress for massmailing • Available in taxpayers files • Questionnaire • Fitted subset of the CAPI questionnaire • Length around 10/15 minutes • Notification letter, reminders • First, a postal mail with a notification letter • 2 reminders (one with paper questionnaire)

5. Examples of experiments • Housing 2010 (Amiel and Denoyelle, 2012) • Comparison with results of Housing 2006 (face-to-face) • Internet respondents less satisfied (calibration techniques) • Safety survey (SASU) 2013 • First experimental survey on victimization • 2 protocols to control the problem of self-selection • Housing 2014 • Focus on rental amount • External documents without the interviewer ? • Wealth 2014 • Focus on value of real estate property • Deviations from market value without the interviewer ?

6. Working conditions Quality of life at work (QLW) • Working conditions survey (face-to-face survey) Survey adresses many topics surrounding working conditions : • working hours and organization of working time • psychosocial constraints • relations with the public … • Experimental parallel survey : Quality of life at work Tackles same topics : shorter version of the questionnaire. Sample drawn in a base built on taxpayers files : only people that have declared earnings (in 2010) from work can be selected.

7. QLW – Aims and protocols • Aims • Comparison of • Response rates between internet and internet+paper • Quality and content of responses between traditional and experimental surveys • Responses between 2 protocols involving different notification letters • Test questions of the future survey on psychosocial risks (2015) • Open questions to raise topics for this survey • Protocols • 2 different models of notification letters (psychosocial risks) • For half of the sample, internet and paper (standard protocol), for the other half, only internet

8. QLW – Response rates At the end of the data collection Subsample 1 - model A letter, internet and paper available as modes of response Subsample 2 - model B letter, internet and paper available as modes of response Subsample 3 - model A letter and only internet available as mode of response Subsample 4 - model B letter and only internet available as mode of response (model B letter insists on psychosocial risks)

9. QLW – Sample vs internet / paper Comparison on sample frame variables

10. QLW – Internet vs paper respondants Comparison on survey variables

11. QLW – Examples of mode effects • Differentiation on responses scale (satisficing) • Control of selection with kernel matching on propensity score • Index of response differentiation (McCarty and Shrum, 2000) • Result : differentiation more pronounced on internet than on paper • Emotional well-being • Control of selection with kernel matching on propensity score • WHO-5 well-being index (de Wit et al., 2007) • Result : emotional well-being higher on paper

12. Conclusion • Mixed modes : adaptations to technological changes • Not to be forced without prior evaluation of quality • Future results of experiments will help to re-design • A standard scheme ? Trade-off with society requests

13. References Amiel, M. H,. and Denoyelle, T. (2012), Enquêtes en ligne : comparaison de modes de questionnement sur le thème du logement, XIèmes Journées de Méthodologie Statistique de l’Insee de Peretti, G., and Razafindranovona, T. (2013), Les enquêtes multimode : multi-solution ou multi-problème ?, 45e Journées de Statistique de la SFDS. de Wit, M., Pouwer, F., Gemke, R., Delemarre-van de Waal, H. and Snoek, F. (2007), Validation of the WHO-5 Well-Being Index in Adolescents With Type 1 Diabetes, Diabetes care, 30. Heckman, J., Ichimura, H., and Todd, P. (1998), Matching as an Econometric Evaluation Estimator, Review of Economic Studies,65. Krosnick, J. (1991), Response strategies for coping with the cognitive demands of attitude measures in surveys, Applied Cognitive Psychology. Vol 5. Lugtig, P. J., Lensvelt-Mulders, G. J., Frerichs, R., and Greven, F. (2011), Estimating nonresponse bias and mode effects in a mixed mode survey, International Journal of Market Research, 53(5). Lyberg L. (2012), La qualité des enquêtes, Techniques d’enquête, 38(2). McCarty, J. A., and Shrum L. J. (2000), The measurement of personal values in survey research: A test of alternative rating procedures, Public Opinion Quarterly, 64.