Scottish Household Survey Review 2005

1. Scottish Household Survey Review 2005 Ewen McCaig McCaig Services Ewen McCaig

2. External Review To assess whether the SHS is fit for purpose now, and to ascertain if, and in what form, it should continue • Survey Purpose • Topic Coverage • Data Issues • Survey Design • Output and Dissemination • Project Management and Control • Future Developments of the Survey Ewen McCaig

3. Scope • Not an evaluation of performance or efficiency • Not an ‘independent’ review - purpose was to support a broader SE review process • Options required, rather than recommendations • Strategic, not detailed • Topic cover to continue to focus on transport, communities and local government • Recognise and support external interests but give priority to SE requirements Ewen McCaig

4. Method • Report to Review Team • Document review • Consultation • Topics and dissemination with users (SE and external) • Design and management with SE stakeholders and users Ewen McCaig

5. Consultation • Intended to give all a chance to contribute: • SE staff, • other public sector, • academics, • voluntary sector • consultants • Email, phone and important SE contacts were face-to-face Ewen McCaig

6. Consultation • Contact list compiled from • Scottish Executive personnel identified by the SHS team • Distribution lists of SHS data and reports • Local authority contact lists • Attendees at the last SHS User Group event • Persons who had requested data or special analysis from the SHS team • The ScotStat mailing list • Persons identified during the consultation by snowballing Ewen McCaig

7. Response • Consultation, not a survey, so response rate is not significant • Total emails sent: 454 • Email or postal return: 107 • Telephone interview: 34 • Face to face interview: 35 • No contact: 33 • Do not use SHS: 15 Ewen McCaig

8. Questionnaire Content • All existing topics had users and additional demands identified • Perception that transport was main focus not justified (13.8 out of c50 minutes) • Local authority topic now less prominent but still relevant • Communities topics would benefit from further consultation and extension • Image of survey was not always clear to users • Possible review of income and travel diary questions • Some feeling that ongoing consultation on topic cover could be improved Ewen McCaig

9. Modularisation • Current interview too long and more material wanted • More material cannot be included if everything covered in every interview • Cannot be expected that identical sample sizes needed for all questions in multi-themed survey • Modularisation means covering selected questions with a part sample - eg if 50% of question time asked of 50% sample, then topic coverage increases by 25% Ewen McCaig

10. Modularisation (cont) • Only way to increase cover - more material moves towards critical mass of cover, strengthens thematic identity, introduces flexibility • More elaboration and some cost implications but better value for money • Some questions would be asked of part-samples but still large numbers by Scottish standards • Main adverse impact on disaggregation of some of the findings Ewen McCaig

11. Modularisation Design • Socio-demographic and topic ‘core’ content in every interview • Length of core determines residual time for topics. Illustrative example: • Average interview length 45 minutes • Length if all questions asked 80 minutes • Core length 25 minutes • Non-core 55 minutes • Non-core time available 20 minutes • Implied mean sample size for non-core questions is 5,700 per year or 11,400 bi-annually • Even 12.5% cover gives an annual sample of 1,875 • Detailed design must follow consultation Ewen McCaig

12. Sampling • Total sample size not covered in review - increases possible but must be practicable • Stratification with minimum sample by LA still seems sensible - allows national reporting and tends to weight sample into areas of low population • Key point is clustering - sample currently clustered in the 23 least urban LAs Ewen McCaig

13. Sampling (cont) • Benefits of unclustering clear: the only potential issue is fieldwork capacity and cost • Some allowance already for cost with sample boost intended to compensate for cluster design effects • Increased cost can be offset against increased sample size required to compensate for design effects Ewen McCaig

14. Benefits of Unclustering • No design effects due to clustering • No geographic gaps or bias in the data • Relative simplicity and transparency • Easy expansion to population totals for different areas • Data available for more geographic areas without bias and complex weighting options • Fewer obstacles and risks for secondary analysts • Easy utility in systems involving geographic selection and analysis • Flexibility in merging samples with other surveys Ewen McCaig

15. Other Issues • Dissemination could emphasise thematic content more • Consultation on topic cover important • More emphasis on web dissemination could be considered • Find role in development of cross-sectional survey standards in Scotland Ewen McCaig