Soames Job Michael Cornish Chika Sakashita Motor Accident Commission

1. Enhancing the Evaluation of Road Safety Communication Programs: Developing an Evaluation Strategy Soames Job Michael Cornish Chika Sakashita Motor Accident Commission Australasian Road Safety Conference, Gold Coast, October 2015

2. overview ABOUT MAC • MAC • Evidence-based Road Safety Communications • MAC Evaluation Strategy • Purpose • What it involves • How to use it • Trial of implementing the Strategy • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

3. Motor accident commission (MAC) ABOUT MAC • CTP provider in South Australia • Road safety communications to various audiences via paid advertising placements, digital media, partnership and sponsorship programs • Fund(ed) road infrastructure work and assessed in collaboration with DPTI to select works • Committed to evidence-based development-implementation-improvement cycle • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

4. Evidence-based road safety communications improvement cycle ABOUT MAC • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

5. Evaluation of road safety communications ABOUT MAC • Critical to understand which communications are working, by how much, and for whom • Not only a success demonstration exercise but also a learning opportunity for future improvements • Not easy or cheap, but good value • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

6. Mac evaluation strategy ABOUT MAC • MAC has developed a practical Evaluation Strategy based on a review of international research and best practice • Process and outcome research within funding limits • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

7. Process evaluation ABOUT MAC • Analysis against a Best Practice Checklist – fundamental to the development of the communication program • Estimation of effects based on Campaign Features – see recent meta-analyses of previous outcome evaluations • Phillips, R. O., Ulleberg, P., & Vaa, T. (2011). Meta-analysis of the effect of road safety campaigns on accidents. Accident Analysis & Prevention, 43, 1204–1218 • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

9. Estimation of campaign effects ABOUT MAC • Assume 12% reduction in crashes if meet all the 15 good core practice criteria (based on meta-analysis) • Conservative allowance of 1% extra benefit if campaign at state level • Conservative allowance of 3% additional benefit if campaign aligned with enforcement activities and the message emphasises the risk of detection, etc., or if not reduce by 3% • No allowance for the emotional content (fear, fun, guilt, sympathy) – not clear all these would work similarly well • Assume beneficial effects last for the campaign duration at a maximum of 3 months • Assume net crash reduction applies equally to all crash severities (no precise information provided by the meta-analysis) • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

10. Outcome evaluation ABOUT MAC • Understanding different types of communication programs is critical to effectively choose the most appropriate evaluation approach • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

11. Communication program type ABOUT MAC Type 1: Influences behaviours directly, sometimes without and sometimes followed by attitude change. Type 2: Influences behaviours via attitudes. Type 3: Generates changes in attitudes or beliefs which allow more effective other actions which will change behaviours e.g. reduced speed limits, reduced BAC limits, increased enforcement or increased penalties • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

12. Types I & ii: 4 tiers of outcome evaluation ABOUT MAC • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

13. Type Iii: 5 tiers of outcome evaluation ABOUT MAC • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

14. Choosing the optimal outcome evaluation tier ABOUT MAC • Consider advantages and disadvantages against budget, timelines, subsequent relevant change of policy, regulation, or practice • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

16. Trial of evaluation strategy in progress ABOUT MAC • MAC’s recent drink-driving campaign ‘Grow Up’ – a novel large-scale program: Outdoor, print, TV • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

17. Trial of evaluation strategy in progress ABOUT MAC • MAC’s recent drink-driving campaign ‘Grow Up’ • Singular focus on drink-driving behaviour • Type I/II • Objective measurement of drink-driving not particularly feasible • Tier 2 outcome evaluation • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

18. Latest MAC ad SENSITIVE: COMMERCIAL

19. acknowledgements ABOUT MAC Road Safety & Strategic Communications, MAC • Michael Cornish, General Manager • Richard Blackwell, Marketing Manager Global Road Safety Solutions • Chika Sakashita, Road Safety Consultant • Responsible for managing South Australia’s Compulsory Third Party (CTP) scheme, providing approximately $400 million in compensation each year to assist road crash victims. • Since March 2007 responsible for road safety education campaigns– approx $10million per year

20. Thank you for your attention!