511 Program Performance Monitoring Evaluation in the San Francisco Bay Area

1. 511 Program Performance Monitoring & Evaluation in the San Francisco Bay Area Presented by Carol Kuester Metropolitan Transportation Commission Session 3B: Evaluating Your 511 Service 511 National Conference July 18, 2006 San Diego, California

2. Presentation Agenda The Bay Area 511 Program Commitment to Performance Monitoring & Evaluation Evolution of Specific Measures Usage Satisfaction Reliability Accuracy On-going Challenges

3. 9 San Francisco Bay Area Counties 7 Million People; almost 4 million jobs 101 Municipalities 1,400 Miles of highway 19,600 Miles of local streets and roads 23 Public transit operators What is the MTC 9 County region?

4. Overview of 511 Bay Area�s free traveler information service Phone � Call 511 Web � Visit 511.org Multimodal Transit Traffic Ridesharing Bicycling 100,000+ calls/week 390,000+ web user sessions/week

5. Commitment to useful and accurate information Mission statement: �The 511 program must cost-effectively provide traveler information that customers want and are prepared to act on�This information should be accurate, reliable, multimodal, comprehensive and regional in scope�.� Strict system performance requirements and award fees used to motivate contractors

6. Performance monitoring factors Requirements vary by mode and by contract Common factors monitored across modes include: Usage (e.g. number of calls, web sessions, and use of specific features) Accuracy (e.g. comparison of reported travel time to ground truth, accuracy of trip itinerary, accuracy of rideshare database entries) Reliability (e.g. system up-time, web site responsiveness) Some additional factors for traffic: Customer satisfaction 24/7 operations center staff adherence to Standard Operating Procedures

7. Usage

8. Over 400,000 calls per month Over 1,400,000 Web user sessions per month Cumulative usage to date: nearly 12 million calls and 25 million Web user sessions Phone and Web Usage Summary As of Th/Fr, milestone reached: 10 million calls. NOTE: Ops memo said over 13 million user sessions; should have been over 17 million user sessions. Need to mention at mtg. Well used today with over 1 million calls/visits per month. Serves region well in times of emergencies. Phone compares well with other regions � most popular in country. Very few metropolitan peers; most are statewide systems. Majority of phone requests are for traffic information, majority of web requests are for transit information As of Th/Fr, milestone reached: 10 million calls. NOTE: Ops memo said over 13 million user sessions; should have been over 17 million user sessions. Need to mention at mtg. Well used today with over 1 million calls/visits per month. Serves region well in times of emergencies. Phone compares well with other regions � most popular in country. Very few metropolitan peers; most are statewide systems. Majority of phone requests are for traffic information, majority of web requests are for transit information

9. 511 Millionth Caller Milestones As of Th/Fr, milestone reached: 10 million calls. NOTE: Ops memo said over 13 million user sessions; should have been over 17 million user sessions. Need to mention at mtg. Well used today with over 1.2 million calls/visits per month. Serves region well in times of emergencies. Phone compares well with other regions � most popular in country. Very few metropolitan peers; most are statewide systems. Majority of phone requests are for traffic information, majority of web requests are for transit information As of Th/Fr, milestone reached: 10 million calls. NOTE: Ops memo said over 13 million user sessions; should have been over 17 million user sessions. Need to mention at mtg. Well used today with over 1.2 million calls/visits per month. Serves region well in times of emergencies. Phone compares well with other regions � most popular in country. Very few metropolitan peers; most are statewide systems. Majority of phone requests are for traffic information, majority of web requests are for transit information

10. Usage goals What level of usage is good enough? Ambitious goals may encourage performance, but what are the ramifications if they aren�t met? No benchmark for percentage of target groups expected to use 511; market penetration goals tough to establish Should the goal be to increase the number of individual users, or uses? How to measure impact of delivering improved data to broadcasters, or benefit of information on changeable message signs? How should usage levels be adjusted as the program matures and new features are added?

11. Traffic Performance Monitoring

12. Revised traffic performance requirements Contract award fee was initially based on usage and satisfaction Pro: Helped ensure that the contractor shared agency goals and worked creatively to improve the system. Issues: What should usage targets be? Satisfaction levels were consistently high Surveys can be expensive, going forward internal web and phone tools will be used User surveys don�t capture those who were so unsatisfied that they no longer call or log on

13. New accuracy and reliability requirements Accuracy Reported vs. actual driving times Reported vs. actual incidents and slowdowns Accuracy of entered incidents vs. CHP reported incidents Timeliness of incident reporting SOP compliance during major incidents (post-incident audit) Reliability Major system failures Less than 45 hours/year = full award, more than 90 hours/year of failure = no award in this category

14. On-Going Challenges

15. Performance Monitoring Challenges Monitoring adherence to requirements is resource intensive For traffic contract, contractor developed a Performance Monitoring Plan and hired dedicated staff for performance monitoring functions In addition, MTC retains a separate performance monitoring contractor Performance monitoring can produce too much information Monitor program elements that can be influenced Tie consequences to factors that are monitored Monitor outcomes rather than adherence to processes

16. Program evaluation: what is good enough? How can we be sure that traveler information has a positive impact on the transportation system? Link to reduced congestion? Traveler peace of mind and/or time savings?