1 / 24

Online Opinions – A Pilot to Extend ONS Social Data Collection Capabilities Ed Dunn

Online Opinions – A Pilot to Extend ONS Social Data Collection Capabilities Ed Dunn ONS Social Survey Division. Overview. Background and context setting The pilot study; our key objectives and design Findings Next steps. Pilot Key Objectives – ‘Dipping our Toes’.

leone
Télécharger la présentation

Online Opinions – A Pilot to Extend ONS Social Data Collection Capabilities Ed Dunn

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Online Opinions – A Pilot to Extend ONS Social Data Collection Capabilities Ed Dunn ONS Social Survey Division

  2. Overview • Background and context setting • The pilot study; our key objectives and design • Findings • Next steps

  3. Pilot Key Objectives – ‘Dipping our Toes’ • Need to examine web collection • Investigate unit and item response rates • Investigate respondent characteristics • Evaluate the hardware/software capabilities Examine the effect of offering an incentive • Design and administration of web surveys • Recommendations for further work • Deliver to a short timescale (by December 2008)

  4. Sample Design and Respondent Selection – Web Pilot • Simple random sample of 2,000 households from English Postal Address File • Advance letter • Half offered a £10 Amazon gift certificate to take part online • Letter asked adult with most recent birthday to complete the survey • Unique username and password provided to access survey from ONS website • Follow-up letter

  5. Sample Design and Respondent Selection- Opinions Survey (OPN) • OPN is multi-stage cluster design • Respondent selection via KISH grid • November OPN sampled 2,010 households • Book of postal stamps sent with advance letter

  6. Based upon ONS Opinions Survey (OPN) Used question blocks equivalent to a ‘normal’ month of OPN Questions reviewed for suitability for web Some modifications necessary Questionnaire tested internally Questionnaire Design (1)

  7. Questionnaire Design (2) • Design and layout reviewed for web survey* • Page Design • Next/Back Button • Progress Bar * See Dillman, D.A (2007) Mail and internet surveys - the tailored design method (2nd edition) J Wiley & Sons, New Jersey

  8. Questionnaire Design (2)

  9. Unit Response Internet pilot • 364 fully complete responses received (18%) • 32 partial responses (2%) • Little difference between incentive (18%) and non- incentive (19%) group • But, Full sample= 2000 10% unknown eligibility and only 61% have internet access at home (LCF) Available sample = 1098 New Response = 33% complete responses November OPN • 59% overall response (with known eligibility)

  10. Metrics • 80% of respondents completed web survey in under 25 minutes • 75% completed 8am-6pm • Most popular hours to begin survey were 4pm-5pm (10%) and 7pm-8pm (11%) • 25% completed on weekend • Wednesdays and Fridays most popular • OPN face to face takes c.45mins (excl. admin time)

  11. Results (1) • Weighted results; OPN routinely weighted but pilot required some methodological work. OPN was then re-weighted to same population totals. • 2-tailed t test applied to differences • Some clear differences in profile of internet pilot respondents compared to OPN and key question responses

  12. Results (2) • The pilot respondents were more likely to be: • Aged 25-44 and 55-64 • Married and living with partner • White • Better educated (with degree level qualification) • Managers or supervisors • In good health

  13. Results (3) • In terms of key question responses pilot respondents were less likely to: • Smoke • Have a disability • Think charities played an important role in society • Think HM Revenue & Customs treated them fairly • Pilot respondents were more likely to: • Eat healthily (5 portions of fruit or vegetables a day)

  14. Next Steps • Limitations of this pilot widely accepted • Pilot has demonstrated viability of conducting a major ONS social survey online and receiving a significant, if minority, response • Pilot has highlighted logistical and technological issues • Pilot has highlighted risk of significant bias • Identified areas for further work via more sophisticated pilot designs • Plan to run a further pilot on OPN in 2010 and pilot on LFS early in 2011

  15. References • Dillman, D.A. 2007. Mail and Internet Surveys – The Tailored Design Method (2nd ed). J Wiley and Sons, New Jersey • Flatley, J. 2001. The Internet as a Mode of Data Collection in Government Social Surveys: Issues and Investigation Social Survey Methodology Bulletin, ONS 49(7) • Fricker, R and Schonlau, M. 2002. Advantages and Disadvantages of Internet Research Surveys: Evidence from the Literature. Field Methods 14(4) • Solomon, D. 2001. Conducting Web-Based Surveys. Practical Assessment, Research and Evaluation 7(19)

  16. Internet and the UK LFSDean FletcherUK ONS

  17. Internet and the UK LFS • Current Modes: • Wave 1 Face to Face Wave 2-5 TU or ftf • Response on UK LFS all time low 65% wave 1, 57% overall. • Achieved sample size decreasing or much more costly to obtain • Efficiency savings face to face costs 3 times TU • Pilot shows some groups with current high non-response will take part in the internet. • TU costs 3 times Internet.

  18. Internet and the UK LFS • Pilot showed technology exists. • 4 main Problems: • Collecting close to the Reference week • Problem of integrating the internet with the other modes to prevent bias • Mode effects • Collecting from Households not persons

  19. Registered for internet Respond to internet FtF W1 TU W2-5 Internet and the UK LFS • Analysed other approaches but derived duel sample approach after considering results of the pilot Initial Sample Face to Face W1 TU W2-5

  20. Internet and the UK LFS • Currently reviewing questionnaire for internet suitability based on pilot results and others research to minimise mode effects. • Household collection seems feasible • Use Confirmit to test ideas • Use BLAISE IS? To run pilot in 2011 in a few NUTS2 areas • If successful run full design by 2012 • Modify collection by 2013

  21. Internet and the UK LFS • Issues • Internet pushes up refusals • Unequal response • Changes to internet over time

  22. Internet and the UK LFS • Also monitoring wave 2-5 work by other NSi’s to implement if shown to be successful • But current systems do not allow internet, TU and ftf integration. • Cost savings and non-response mainly achieved for UK at Wave 1.

  23. Internet and the UK LFS • So far we have had: • OPN Pilot • Input at International Workshop • Prof. Mick Couper’s work and training • Literature review • But interested to have design scrutinised for issues.

  24. Internet and the UK LFS • Concluding remark • UK Can’t afford not to implement an internet collection mechanism if possible in the next 2 years.

More Related