NCHRP 20-24(37C) – Comparative Performance Measurement

1. NCHRP 20-24(37C) – Comparative Performance Measurement Safety Performance Based on the Fatality Analysis Reporting System (FARS) AASHTO Safety Management Subcommittee • September 3, 2009 • Hyun-A Park

2. Content • Project Context • Why comparative performance measurement? Why is it important? • History • Project Objectives and Scope • Summary of Findings • Data Analysis Approach • Identification of Top Performers • Identification of Practices Leading to Good Performance • Top Performing States • Safety Best Practices (Noteworthy State Practices) • Safety Data for Comparative Performance Measurement • Value of Comparative Performance Measurement for Safety

3. Project Context - Why Comparative Performance Measurement? • State DOTs share similar strategic goals • Best practices can be shared – learn more from each other • DOTs’ senior executive staff seek means for understanding & learning from differences • Counter outside efforts to measure DOTs (e.g. TTI, Hartgen, Governing Magazine) • New tool for implementing continuous improvement philosophy • Provide “one-stop shop” for priority and emerging business areas (e.g. safety, project delivery, asset management)

4. Project Context - Why Is It Important? Growing Importance of Performance Management in Government Transportation • Next federal transportation bill is likely to have a requirement for relating performance to the budget • Each comparative performance measurement project has revealed the weakness of the current data set for conducting comparisons across states • This program has been embraced by the new AASHTO Standing Committee on Performance Management • Many senior executives (CEO’s and Deputies) are members of SC • Need to raise awareness across states and accelerate improvements in policies, data, and behavior

5. Project Context - History • Comparative performance measurement effort for state DOTs initiated in 2004 • Initial set of activities involved workshops and conversations with executives to identify candidate areas of interest and pilot focus area • Project delivery chosen as pilot focus area. Seven states sign on for pilot - Delaware, Florida, Missouri, New Mexico, Ohio, Virginia, and Washington State • Developed prototype approach for comparative performance measurement • Pilot project transitioned to first comparative performance measurement study – on-time and on-budget construction • Study conducted in 2006, report completed in March 2007 • Second project for smooth pavements (IRI) was started in September 2007 and finished in April 2008 • Project for safety comparative measurement was started in September 2008 and completed in spring 2009

6. Construction Project Cost and Schedule On time/on budget performance: • 20 states • 5 years • 26,000+ projects Used to id & profile best practices for achieving results

7. Smooth Pavements – International Roughness Index • Practices used by top performing states require a clear focus by the agency, and policies and programs that support that focus. • Highlights identified include (1) use of end result ride specifications with financial incentives for good performance and (2) establishment of close working relationships with the contractor community. • Five agency practices and four contractor practices were identified as valuable for achievement of smooth pavements. • Recommendations for improving future comparative performance measurement Using IRI developed

8. Summary Findings – Safety Performance • Improvements in safety performance were the result of coordinated, focused efforts on the 4E's (engineering, enforcement, education, and emergency response) with solid leadership, committed players, and dedicated resources - no big surprises, no silver bullets • High degree of consistency across states interviewed in what people felt was important Key factors leading to good performance were: • Data-driven decision making from top to bottom (at both strategic and tactical levels), • Seat belt and DUI legislation and enforcement involving a large percentage of local law enforcement agencies, • Close working relationships across key players at state and local levels (and as a corollary, a unifying umbrella or brand for all safety efforts), • Effective use of media (supported by considerable outreach and funding) • Use of low-cost engineering improvements at state and local levels

9. Study Scope - Background Context • AASHTO Board of Directors has set a goal of reducing fatalities by 1,000 per year over the next 20 years to cut fatalities in half. • Several states have set their own aggressive targets for fatality reduction, and have shown success moving towards these targets • Many safety performance measurement initiatives underway • Safety performance responsibilities do not reside only in state DOTs • Fatality Analysis Reporting System (FARS) is the only national database for safety-related data that does not rely on a sampling approach

10. Study Scope - Background Project Goal • Identify states with best safety performance • Who has achieved greatest improvements in safety performance within the past five years • Peer group states to make most meaningful comparisons • Identify practices used by top performers • Share effective practices with all states • Make information on methodology, analysis, and results available to all who could benefit from the study

11. Study Scope - Approach Original Proposal: • Use similar methodology used for smooth pavements/IRI for this study • Began with participating agencies to determine the measures and associated data to be used for comparative performance analysis • Use participants states input to determine performance measures and data components • Developed template for states to submit their data • Determined top performing states • Interviewed agencies for best practices • Present best practices

12. Study Scope - Approach Shift in approach required given differences between Safety and the two prior efforts • Not able to generate participation of significant number of states because safety performance does not reside solely under state DOTs • Fatalities (FARS) selected as the primary indicator given importance to states and availability of reliable data • Existence of multiple causal factors and the fact that fatalities are rare events pose challenges in drawing clear links between agency actions and performance results. • Implications: • There may be states with safety practices that have proved to be very effective in a site-specific context that won’t be picked up through this analysis -- supplemented study results with literature review findings • Exogenous factors that may account for some of the improvement observed in the states identified as good performers – identified these in the analysis

13. Study Scope - Approach Specification of performance measures • Used available FARS data to analyze state safety performance: 2000-2007 • Used FHWA VMT figures for urban and rural classified roads – by state for 2000-2007 Data analysis and selection of top performing states • Determined peer grouping factors used for analysis • Conducted analysis of fatality rates by state to identify top performers • Used four different methods • Identified top performers • Interviewed candidates • Used existing documents and recent case studies Determining Top Performers Using FARS Data Determined practices contributing to fatality reductions

14. Safety Performance Indicators • Total fatalities • Fatalities/VMT - total • Rural fatalities/rural VMT - based on functional class • Urban fatalities/urban VMT – based on functional class • Number of drivers aged 20 or younger involved in fatal crashes • Number of unrestrained passenger vehicle occupant fatalities, all seat positions • Number of fatalities in crashes involving a driver or motorcycle operator with a BAC of .08 and above • Number of speeding-related fatalities • Number of motorcyclist fatalities • Number of un-helmeted motorcyclist fatalities • Number of pedestrian fatalities

15. Final Set of Performance Measures • Primary Performance Measure • % change in 3 year average between 2000-2002 and 2005-2007 in total fatalities per 100 million VMT • Secondary Performance Measure (tie breaker or screening) • 3 year average 2005-2007 Fatality rate • Supplemental Performance Measures • % change in 3 year average between 2003-2005 and 2005-2007 in total fatalities per 100 million VMT (most recent portion of the 2000-2007 time period) • % change in 3 year average between 2000-2002 and 2005-2007 in urban fatality rate (fatalities on roads with urban functional classification divided by 100 million VMT on roads with urban functional classification) • % change in 3 year average between 2000-2002 and 2005-2007 in rural fatality rate (fatalities on roads with rural functional classification divided by 100 million VMT on roads with rural functional classification) • Change in 3 year average of total number of fatalities

16. Methods for Identification of Top Performing States Four methods used – recommendations reflect “union” across methods • Straight ranking based on percentage change in fatalities • Screen out states with lower than national average fatality rate, then rank based on percentage change in fatalities • Geographic peer groupings - select top state within each of 5 geographic zones (see next slide) – based on percentage change in fatalities, with absolute fatality rate as tie-breaker • Urban/Rural peer groupings – 5 groups based on percentage of 2000-2002 fatalities on urban classified roadways – select top state within each group – based on percentage change in fatalities, with absolute fatality rate as tie-breaker

17. Peer Groupings - Geographic Regions • Used by FHWA for Travel Monitoring – 5 regions (http://www.fhwa.dot.gov/ohim/tvtw/08maytvt/region.cfm)

18. Peer Groupings - % of Fatalities on Urban Roadways • Group 1 • Less than 15 percent of fatalities on Urban Classified Roads (9 states) • ME, MS, MT, ND, SC, SD, VT, WV, WY • Group 2 • 16-30 percent of fatalities on Urban Classified Roads (15 states) • AL, AS, ID, IN, IA, KS, KY, MN, NE, NH, NM, OK, OR, UT, WI • Group 3 • 31-45 percent of fatalities on Urban Classified Roads (14 states) • AK, CO, DE, GA, LA, MI, MO, NC, OH, PA, TN, TX, VA, WA • Group 4 • 46-60 percent of fatalities on Urban Classified Roads (8 states) • AR, CA, FL, HI, IL, MD, NV, NY • Group 5 • Over 60 percent of fatalities on Urban Classified Roads (4 states) • CT, MA, NJ, RI

19. RESULTS – Method 1 Five States with Largest Fatality Reductions • The five states with the largest percentage reductions in their fatality rate are listed below • These states had reductions of three to four times the rate for the country as a whole • Country average is seven percent reduction in fatality rate between 2000-2002 and 2005-2007

20. RESULTS – Method 2 Five States with Largest Fatality Reductions of the States with Fatality Rates Lower than the National Average • Two step analysis • First, 25 states with fatality rates lower than the national average of 1.41 were selected • Then, these 25 states were sorted by the percentage change in fatality rate. • The five states with the greatest fatality reductions are listed below. • Both methods 1 and 2 result in selection of Colorado, Michigan, Utah and Minnesota. When states with fatality rates higher then the national average are eliminated, Connecticut replaces Alaska in the list.

21. RESULTS – Method 3 Geographic Peer Groups • Peer groupings were based on the five FHWA regions used for presentation of travel monitoring information. • Rankings within peer groups were conducted based on percentage reduction in fatality rate. • Overall fatality rate was used as a tie-breaker. • For two of the regions, a second state was identified as worthy of consideration for investigation.

22. RESULTS – Method 3 (continued) Geographic Peer Groups • Performance results for selected states (including the two “runner up states”) are shown below: • Using the geographic peer grouping method adds TX, MD and MA into the group of states for consideration.

23. RESULTS – Method 4 Peer Groupings Based on Share of Fatalities on Urban Roadways • The peer groupings were assembled based on the share of total fatalities on roadways classified as Urban – from 2000-2002 FARS data (see list of peer groups on slide 13)

24. RESULTS – Method 4 (continued) Peer Groupings Based on Share of Fatalities on Urban Roadways • Performance results for selected states are shown below: • This peer grouping adds two new states – ME and NY for consideration.

25. RESULTS – Method 4 (continued)

26. Performance Results

27. Alaska Colorado Connecticut Maine Maryland Massachusetts Michigan Minnesota New York Utah Selected States Top Performing States - Interviewed Top performing with multiple methods

28. Summary Results and Recommendations for Best Practice Identification * tied or close second

29. Existing Safety Performance Measurement & Comparison • NCHRP Project 17-18 (016) – Creating a Traffic Safety Culture – A Case Study of Four Successful States • Case Studies of Iowa, Minnesota, Michigan and Washington • Importance of strong cross-agency partnerships, champions, statewide reach • NHTSA & GHSA report (Hedlund, Aug 2008) - agreed on minimum set of safety performance measures for states and federal agencies – used for this effort • FHWA/AASHTO/NCHRP/Austroads International Scan Report - Halving Roadway Fatalities (April 2006) • Performance-based process including problem identification, benchmarking, target setting, strategy identification, monitoring, and integration of results into future planning • FHWA Comparative Performance Report – used similar methodology – compared 3 year average fatality rates 1996-1998 to 2002-2004 • Success states identified: IA, MI, UT, WA, and OR – 18-25% reductions as compared with 9% reduction nationwide • Success states used multi-faceted approach to safety improvement • Report also reviewed results from four international scan tours: major success factors were speed management (including road design to achieve speed conformance), enforcement, public education, data collection, and top-down leadership

30. Existing Safety Performance Measurement & Comparison • Making the Case for Transportation Safety (FHWA - September 2007) • Case studies of best practices with documented impacts • NCHRP 20-24(44) Highway Safety Leadership Organizational Issues: A Survey of States and Recommendations for Sustaining Progress • Importance of safety champions, no “one size fits all” model. Safety Office close to governor’s office or in independent capacity can be most effective model, but success is dependent on right balance of internal leadership, legislative interest, and public support • AASHTO Safety Leadership Forum (2007) • Successful programs of three states winning AASHTO Safety Leadership Awards - IA, MI and WA • Programs emphasized obtaining and widely distributing traffic accident data and analysis tools, promoting broad-based collaboration among state and local agencies, and focusing on an aggressive statewide safety goal

31. Interview Topics • Performance Management Approach: goals and targets established, focus areas, use of performance data to guide strategy and countermeasure development • Resource Allocation: allocations to specific safety programs, incorporation of safety considerations into planning and programming processes • Collaboration and Partnerships: mechanisms for interdisciplinary, multi-agency and interjurisdictional coordination; coordination within the DOT across traffic and safety engineering, design, construction, planning and programming functions • Legislation: passage of legislation related to passenger restraints, helmet laws, DUI, cell phone use, graduated driver licensing • Engineering Solutions: systemwide programs (e.g. for lane departure reductions), black spot programs, use of road safety audits, methodologies for targeting and prioritization of countermeasures to make best use of resources • Enforcement Solutions: use of targeted or data-driven approaches, technology applications, synergistic programs combining education and enforcement • Education and Technical Assistance: effective methods for changing driver behavior (speeding, aggressive driving, seatbelt use, helmet use, impaired driving) use of media campaigns alone or in combination with other solutions, training and technical assistance targeted at local agencies • Emergency Response System improvements • Demographic/other “exogenous” changes contributing to fatality reduction

32. Practices Improving Highway Safety Investment Engineering Safer Roads Partnerships Data Communications

33. Safety Best Practices • Provide strong leadership • Establish an executive level committee to ensure coordination and cooperation and guide major resource allocation decisions • Identify and empower safety “champions” including state and local elected officials, safety program managers, police chiefs, and other community leaders. • Keep the collective “eye on the ball” • Set targets through strategic safety planning processes and actively monitor progress • Continually share performance results among major safety partners • Improve timeliness of crash data through automation of police reports • Distribute maps of crashes by type • Make current crash data accessible on the web

34. Safety Best Practices (continued) • Support legislation to reduce highway fatalities • Primary seat belt laws to allow for citations to be issued for lack of seat belt use alone • Graduated driver licensing to impose restrictions on younger drivers while they gain experience • .08 blood alcohol concentration per se laws enabling immediate license revocation for drunk drivers • Cell phone usage laws to reduce distracted driving • Invest in low-cost engineering improvements with demonstrated effectiveness • Shoulder and median rumble-strips • Cable guardrail/barrier treatments • Curve delineation/signage • Signalization

35. Safety Best Practices (continued) • Use data to target engineering, enforcement, education and emergency response programs where they will have the greatest payoff • Use the Strategic Highway Safety Planning process to identify emphasis areas and select appropriate strategies. • Use crash data to guide allocation of resources across program areas to target crash types or behaviors that account for a large share of fatalities. • Conduct screening of locations for engineering countermeasures based on crash and highway inventory data. • Focus enforcement activities on jurisdictions, highway locations and time-periods where the highest concentrations of targeted crash types exist. • Community-Based Outreach – target outreach programs to specific populations based on overlaying census, driver licensing and crash data to identify residential areas and demographic segments with higher than average risks.

36. Safety Best Practices (continued) • Adopt a unifying message for all agencies with a highway safety mission • Agree on a “mantra” or theme (e.g. “Towards Zero Deaths” or “Zero Fatalities”) and brand all safety programs and safety-related messaging with this theme • Promote the understanding that everyone is working towards a common goal, and that collaboration across agencies is the only way to succeed. • Ensure that there is regular communication among safety partners to share information about problems and their solutions • Maintain weekly informal communication among lead staff for engineering, behavioral programs, and state police • Ensuring involvement of a broad set of staff in both strategic and operational decision making to support continuity in programs • Maintain two-way information flow between enforcement personnel and central safety office personnel to share information about problems and recommended solutions

37. Safety Best Practices (continued) • Maximize coordination across state and local law enforcement agencies • Use law enforcement liaison(s) (LELs) to maximize involvement of local enforcement agencies in statewide initiatives • Fund (through federal and state sources) overtime enforcement and equipment purchase, with “strings” that support resource sharing • Develop resource sharing agreements among law enforcement agencies • Pursue creative and proactive public communications and messaging • Develop and maintain strong relationships between the Highway Safety Office and local media • Involve political leaders in safety-related events to provide enhanced media coverage • Maximize opportunities to obtain “earned media” coverage (press releases, events) • Ensure local media coverage of each fatality • Maintain dedicated in-house staff or contractor support to handle communications for all safety messaging • Develop community-based messaging (e.g. Spanish language, church-based, school-based programs)

38. Safety Data for Comparative Performance Measurement • Shift to use of fatalities to fatalities+injuries to provide a more robust basis for comparison • Fatalities are relatively rare events – random variations impact performance results • Continue work on consistency across states on definition and reporting of serious injuries. • Support improvements to timeliness of both crash and VMT data • States that have achieved quick turnaround have reported significant benefits in terms of willingness to use the data to target resources where they will have the greatest payoff • Improve locational accuracy of crash data (particularly for local roadways) • Quantification of both enforcement activities and engineering improvements to allow for cross state comparison would be of value • % of freeway miles with shoulder rumble strips by year (for engineering) • % of annual nighttime/weekend VMT represented by enforcement activities.

39. Value of Comparative Performance Measurement for Safety • Role of this study was to highlight effective practices associated with states that “moved the needle” for traffic fatality rate during the 2000-2007 timeframe • Results will add to a growing compendium of best practices for multiple important dimensions of DOT practice (construction delivery, pavement smoothness, safety…) • Bridge condition and incident response this year • Comparative performance approach provides a compelling basis for executives to understand the potential for further improvement and the practices to be explored for achieving that improvement. • Linking results to practice at a macro level in the case of safety has proved valuable not so much for discovering new practices, but for reinforcing and lending further support to already recognized best practices.

40. Value of Comparative Performance Measurement for Safety (cont.) • Looking at what multiple high performing states have done has allowed distillation of important practices from what is a highly complex and multi-faceted endeavor • Compared to the two other areas (on-time, on-budget and pavement smoothness), safety is more "mature" with respect to performance measurement and use of performance data to target improvements. • A single "take away" from the interviews with top performing states, it is the critical importance of being able to FOCUS activities based on credible and timely data. • Safety practitioners have institutionalized use of performance data for discovery of what works well and what doesn't. • Continued work towards improving comparability of safety data across states is important - it is likely to provide considerable value by enabling states to learn more from their peers.