NDC Household Survey 2006

1. NDC Household Survey 2006 An overview Rachel Williams, Jessica Vince and Leon Page

2. Session objectives • Background information about this year’s household survey • Review data outputs available and give practical tips on reading and interpreting data • The Extranet (www.ipsos-mori.com/ndc) • Sources of help at Ipsos MORI

3. Background to the household survey

4. Overview • Survey aims to provide information on the key outcomes from the NDC programme and to show change over time • Third survey in the series • initial survey in 2002 • 2004 and 2006 surveys are longitudinal – going back to as many of the original 2002/4 respondents as possible plus a cross-sectional top-up sample • Ipsos MORI/GfK NOP conducted 15,792 interviews between May and October 2006 – c.400 in each NDC area • National benchmarks - from existing surveys and new Omnibus survey – are NDC areas ‘catching up’?

5. Changes for 2006 • Sample size reduced to 400 interviews • Longitudinal and top-up addresses issued in the same proportion as 2004 • Questionnaire length reduced: • 25 minute core questionnaire • Partnership specific questions optional • No research among movers, beneficiaries or business

6. Questionnaire

7. Core questionnaire – asked in all NDC areas • Core questionnaire covers each “theme” in national evaluation • Housing and the local environment • Crime • Worklessness • Education • Health • Plus profile information, basic income questions and general perceptions of area/community

8. Core questionnaire – 2 • Wherever possible questionnaire keptidentical to 2002/04 to allow changes in views to be compared • Designed with NET – following outcome review and review following each wave • Full pilot in similar regeneration areas each wave • Some questions ‘follow-up’ answers given by respondents in 2004 e.g. Last time you were very worried about being burgled now you are not very worried – why have your views changed? • Includes national benchmarks to compare against the country as a whole • This year each partnership could add 5 minutes worth of their own questions

9. Sampling

10. Sampling is making an inference about a… Population… …from a… …Sample Why do we sample? • Not feasible to question all residents of a particular area – e.g. 17,000 residents in the East Brighton NDC area alone • Interview a smaller number on the basis that they share the same characteristics as the population generally • Works but relies on strict procedures

11. Bigger is better…but size isn’t everything Sample size

12. Longitudinal sample Return to respondents interviewed in 2004 Top-up sample New addresses selected at random to top up to 400 per area Household survey – two samples

13. Longitudinal sample Longitudinal Return to individuals interviewed in 2004 and attempt to interview them If named respondent has moved or died then randomly select a substitute 10,770successful interviews at original addresses

14. Random selection of addresses from within each of the NDC partnerships Random selection of one property/dwelling or household unit at each sampled address Random selection of one adult within each selected household (5,022successful interviews) Top–up sample Top-up sampling involves a number of steps… Means every person has an equal chance of being interviewed

15. EXAMPLE – Salford NDC 407 interviews in total – 238 longitudinal and 169 top-up Select 237 new addresses - 124 new respondents 400 original 2004 addresses sent out Sample structure - example Interviewed 283 people at these addresses 238 original respondents 45 new respondents

16. Fieldwork

17. Fieldwork • Fieldwork May - October 2006 • Full briefings for all interviewers (28 sessions) • dummy interviews before starting work • where possible using the same interviewers as 2002/4 • Advance letter and FAQ sheet sent to respondents • Interviews conducted using Computer Assisted Personal Interviewing (CAPI) • Minimum of 6 calls, including two at weekend or evenings, plus further re-issues. • Quality control - back-checking and accompaniment • Over 370 interviews conducted in other languages • Response rates improved on 2002 and 2004

18. Household survey outputs

19. Household survey outputs • Range of outputs provided by Ipsos MORI team • Key indicators (most accessible) • Topline questionnaires • Computer tabulations (simple sub-group analysis) • SPSS dataset (most ‘technical’) • Will suggest how and when you might want to use these • Extranet

20. What are key indicators? • Show summaries of some key questionsfrom the survey • Easy to read if showing to non-researchers in your NDC • Useful if you want a quick overview of the results and change over time • Can be used to create a summary of findings for internal use e.g. Board meetings etc. • Cover all the theme areas (Crime, Housing etc)

21. What are key indicators? Columns giving partnership and aggregate data for all three waves, and national benchmark Theme heading Summary of all those very/fairly satisfied with accommodation. Does not show % dissatisfied

22. What are toplines? • Show all of the questionsfrom the survey in the order these were asked on the questionnaire • All responses to each question are shown (e.g. very and fairly satisfied/dissatisfied) not just summaries • Questions are grouped by theme to make them easier to find – e.g. CR questions (CR1, CR2 etc) are from the Crime theme • Use the marked-up questionnaire if you want overall results but do not want to analyse by sub-groups e.g. men vs. women • Fairly easy to read for those not experienced in research

23. What are toplines? Theme heading and question number Full question text List of all responses to the question

24. What are toplines? Filter – an indication of who is asked the question Base – a definition of who was asked the question and how many people this includes National source – most recent benchmarking figures available (mostly for England)

25. Toplines - Interpretation • Number of pitfalls to be aware of when reading and interpreting toplines: • Check the bases underneath the question – is everyone being asked or not? • Some demographic questions are based on all household members rather than all respondents. Gives more accurate information on age, gender and work status • Read the introductory page!

26. Toplines - functionality • Topline document is now in Excel rather than Word • Advantages • Display only question text • Display only benchmarked questions • Quickly calculate change between years using formulas =IF(H17="*",0,H17) etc • Run charts from the data • Will be available in PDF version • Working on formulas to produce automated significance testing (TBC)

27. Statistical reliability - 1 • Not all differences between 2002, 2004 and 2006 data will be significant • E.g. if satisfaction with area has increased by 2 percentage points between 2004 and 2006 this will not mean that there has been a significant increase in satisfaction • Whether changes are significant also depends on how many people were asked a question • The fewer people asked a question, the greater the difference will have to be for this to be significant • E.g. more difficult to find significant difference where 50 respondents have been asked compared with where 400 have been asked

28. Statistical reliability - 2 • Where all respondents have answered a question need differences of: • 8 percentage points or more between 2002, 2004 and 2006 figures to be sure they represent actual change and are NOT due to chance • 6-7 percentage points between individual NDCs and the aggregate or national benchmark figures • Can use our “ready reckoner” – available on the Extranet • An example…

29. Statistical reliability calculator Testing change in proportion feeling unsafe between 2004 and 2006 for Partnership X

30. What are computer tabulations? • All responses to the survey are broken down by around 40 different sub-groups • Sub-groups are respondents who are grouped together in the data because they have something in common. This may be demographic (e.g. their age) or attitudinal (e.g. they want to move out of the area) • Examples of sub-groups are: • Men (demographic) • Those who earn less than £100 per month (demographic) • Those who have heard of NDC (attitudinal)

31. What are computer tabulations? • Why would you want to use these? • You may want to see if all sections of the population are benefiting equally from NDC • E.g. fear of mugging may be going down overall, but are any groups being left behind? • Can compare 2006 results with 2004 results – young people may be 2% less likely to be mugged than in 2004 compared with a drop of 15% overall • Can also use for baseline/monitoring data – are Black residents more likely than average to be earning £100 a month or less?

32. What are computer tabulations? Question on tenure – who are most likely to be private renters? 16-34 year olds significantly more likely to rent privately (text shown in bold)

33. Only 13 people in overall base Can’t compare refugee status of men vs. women What are computer tabulations? BUT: Crucial not to compare sub-groups with small bases as these are unreliable

34. Results for male respondents are in column “a”, females in “b” “x” = total column Computer tables - significance testing • Letters used to identify which results are significantly different from each other

35. More in column “b” (females) and column “d” (35-44 year olds) are lone parent families than average (i.e. column “x”) Fewer in column “a” (men), column “e” (45-64 year olds) and column “f” (aged 65+) are lone parent families than average Computer tables - significance testing • Each column is tested against the total to see whether the differences in results are significantly different • a letter under a result indicates it is

36. A few things to watch… • Some tables run over two pages because the list of possible answers is so long – i.e. 4 pages per question (e.g. WO1) • As well as the actual answers that can be given, also include derived summaries at the end of the list (eg WO1 ILO Unemployed) • There may be tables for the same question but with different bases. E.g. at WO1 there are tables based on all respondents and also on all working age respondents • Sometimes have ‘summary tables’ where a battery of questions are asked • There are a few derived tables e.g. HO3 cross-tabulated by HO4

37. SPSS data • A software program that allows you to conduct your own analyses (Statistical Package for the Social Sciences) • Allows you to do a wide range of analyses that are not included on the Computer Tables • E.g. You could find out the characteristics of ‘Trapped residents’ – those who want to move but do not think they will, by cross-tabulating two survey questions against each other • Every NDC has their own dataset • File contains some additional derived variables eg workless households • BUT – your NDC may not have bought SPSS software so you may not be able to use this data (LA may have it though …) • Ask the Ipsos MORI team if you need any help with using SPSS

38. A note on weighting • Two sets of weights applied to the survey data • Selection weighting to correct for unequal selection probabilities (every person should have equal chance of selection) • Profile weighting to ensure that survey is representative of the population as a whole (based on Census) • All data presented in the survey outputs is WEIGHTED • EXCEPT, the final column on the computer tables - this presents the raw data • Correct weights must also be applied to the SPSS data • NB weighted samples are less accurate – need to bear in mind when checking for significant differences

39. Extranet

40. Extranet - 1 • You may have used this with the 2002 and 2004 data • Analysis tool taken off-line recently to add the new 2006 data • Can still access survey documentation • 2006 data will be available in February next year

41. Extranet - 2 • Extranet allows you some of the flexibility of an analysis programme like SPSS but easier to use • Can analyse your partnerships’ results by demographic factors such as age or gender • Can compare your results against those of other relevant partnerships and against the aggregate data set • Can filter results e.g. the findings just for women aged 16-24 • Can compare change in data over time

42. Extranet - 3 • An on-line resource containing all NDC data • Run your own tables on issues of interest • Download toplines, key findings and computer tables • Data available for all partnerships • Possible to compare yourself with other “local” NDCs or those with similar characteristics • Each partnership has their own log in • Benchmarking data and links to useful sites • Information about the household survey • Background to each wave • Glossary of research terms and training slides

43. Some on-screen examples

44. Using the data

45. Using the data • Background information/reality check along with other sources of data • eg if feelings of safety have increased how does this relate to crime statistics/what the police are saying locally.  • Partnership specific questions used to evaluate specific projects • Post coding data to provide neighbourhood level information • Wider area benchmark surveys

46. Thank you For further information or help please contact: jessica.vince@ipsos-mori.com (020 7347 3148) leon.page@ipsos-mori.com (020 7347 3152) hayley.muller@ipsos-mori.com(020 7347 3252)