Evidence based policy: Presenting statistical information for policy makers

1. Evidence based policy:  Presenting statistical information for policy makers 14 March 2014 Sandra Pattison & Patrick Lim

2. What is the VET policy environment? • Australian Constitution • National, State, Regional, Industry needs • Government funding 2012 • $8428.5 million • $4 333.4 million states and territories • $2 325.4 million Aust government • Private markets, market failure, substitution.

3. What are the key policy questions? • Objective • A productive and highly skilled workforce enabling all working age Australians to develop the skills and qualifications needed to participate effectively in the labour market and contribute to Australia's economic future; and supports the achievement of increased rates of workforce participation.

4. Outcomes • the skill levels of the working age population are increased to meet the changing needs of the economy; • all working age Australians have the opportunity to develop skills; • training delivers the skills and capabilities needed for improved economic participation for working age Australians.

5. How do policy makers like their information presented to them?

6. How do policy makers like their information presented to them?

7. How do policy makers like their information presented to them? vs

8. How do policy makers like their information presented to them? vs

11. Qualifications Students Data Sources? VET Financial Data Apprentice & Trainee VET Provider Employer Survey Student Outcomes Survey Courses/units Characteristics Intentions Outcomes Activity Providers Employers Characteristics Plus other relevant data (eg: ABS Survey of Education and Training, Census, HILDA) Employer views Resources Funding

12. Data Quality • Data quality is vital. • Quality is an on-going and continuous process • Import and collection. • Robust quality assessment when reporting. • Internal and external review of reports and publications.

13. Data Quality on collection • Administrative Data: • Governed by data standards (AVETMISS) and data input tools that ensure consistency with the data standard. • Quality checks undertaken by NCVER upon receiving of the data. • Survey Data: • Close cooperation with the market research companies • Define ranges of acceptable responses for quantitative data. • Quality checks undertaken upon receiving of the data.

14. Data Quality for reporting • Usual exploratory data analysis. • Cross-checking of tables against known bottom-lines. • Research papers - traditional academic referee process: • 2 referees (either internal or external).

15. Defining the question? • Often NCVER is asked by Government to assess a broad general statement: • eg: Has the Productivity Places Program (PPP) worked? What does this actually mean? • We spend considerable effort in helping refine questions. • Data is collected with no specific policy questions in mind.

16. Defining the question? • Often, there is no robust, reliable data to adequately answer a question. • It is “our” role as the key agency in VET to identify weaknesses in data collections and to advise on how to collect the required data/information. • Policy questions can and should inform data development & collection. However, most data collections and surveys exist longer than policy. • Thus, changes to standards/questionnaires needs to be undertaken systematically. • Evaluation needs to be considered when setting up a program, not once it has been delivered.

17. Informing policy? • Very difficult to measure the impact of research, even more-so when trying to determine if it informs policy. • Difficult to attribute a research project to a particular policy development. • Academic journals struggle to inform policy. • Often - unpublished reports have the biggest impact.

18. Informing policy?

19. Informing policy?

20. Layering the dissemination Synthesised into a very short briefing notes for Ministers, Advisors or other parliament committee Presentations Face-to-face debriefs Journal Articles Short summaries Webinars/podcasts Very complicated piece of research may often be reduced to two or three lines in a ministerial briefing

21. Informing policy? • Robust and accurate statistical methodology using reliable data is vital! • Policy makers don’t care that you have fitted a particular type of model. • They need to have faith in the data and methodology. • It is up to us to highlight shortcomings, but this needs to be done succinctly and clearly.

22. Informing policy? • Apprentice and trainee incentive changes • Completion rates • Indigenous systematic review • Part-time work and study • School Effects • Youth Allowance

23. Problems 35 Completion rate = 35% 100

24. Estimating course completion rates Commencing Continuing Commencing Continuing ? ? ? ? ? ? Quit Completed 1 1 Quit Completed Completed Continuing Commencing Continuing

25. Markov Chain model 0.3 0.4 Commencing Continuing Commencing Continuing 0.2 0.2 0.1 0.6 0.4 Quit Completed 1 1 Quit Completed

26. Probability of eventually completing p1 commence continue or p3 commence continue q3 q1 quit complete quit complete

27. Informing policy? • Key Message: Our research won’t influence anyone if policy makers can’t understand what it is we are presenting!

28. Discussion • Evidence base policy – does it exist? • Presenting complicated statistical models (yes, the most basic regression is complicated to others)? • Preserving statistical rigour - how do we ensure that the information presented remains accessible? • How do we balance the need for providing timely and relevant information/data when: • Data sources may be scarce or unreliable • We need to maintain independence • Data Quality Statements – Fit for purpose? • Accepting the fact that your analysis or work may in fact be misrepresented or misunderstood and be prepared for it?