Travel Model Improvement Program Model Validation Improvement Efforts

1. Travel Model Improvement Program Model Validation Improvement Efforts Transportation leadership you can trust. presented toTransportation Planning Applications Committee (ADB50) presented bySarah Sun Federal Highway Administration David KurthCambridge Systematics, Inc. January 13, 2009

2. Travel Model Improvement Program (TMIP)Goals • Conduct outreach and education to advance the state of the practice • Promote research on modeling • Improve model quality

3. Travel Model Improvement ProgramOutreach • Training • Disseminating ideas • Engaging stakeholders • Facilitating research implementation

4. TRB Special Report 288 • http://onlinepubs.trb.org/onlinepubs/sr/sr288.pdf

5. TRB Special Report 288 – Model Practice Shortcomings • Too frequently found • Inadequate data • Optimism bias • Quality control • Validation errors

6. TRB Special Report 288 – Summary Recommendations • Develop and implement new modeling approaches • Ensure better practice • Federal, state, regional collaboration

7. Travel Model Validation Practices Peer Exchange – May 9, 2008 • In response to the recommendation of TRB Special Report 288 • take steps to ensure better practice • Sponsored by FHWA through TMIP • Project manager – Sarah Sun • Project staff – CS and TTI • Exchange participating experts • 9 from public agencies • 2 from academia • 4 from consulting • 4 from Federal agencies (FHWA & FTA)

8. Peer Exchange Objective Use the experience of each panel member to provide practical advice and “action items” useful for moving the validation of travel models closer to “What it should be.” “…What should it be?”

9. Summary of Current MPO Practices • Model Component Validation • Independent data sparse • Estimation and validation using household survey data • CTPP used for home-based work validation • Overall Model Validation • Traffic counts • Transit boardings • Focus on statistics: R2, % RMSE, % Error, VMT Error… • (Documented) Sensitivity Tests Lacking • Few temporal validations

10. Summary of Current MPO Practices • Guidelines and “Standards” Used: • Calibration and Adjustment of System Planning Models (1990) • TMIP Model Validation and Reasonableness Checking Manual (1997) • NCHRP Report 365 (1998) • NCHRP Report 255 (1992) • Various state guidelines

11. Peer Exchange SummaryAssessment of Current Practices • Unwarranted focus on attaining standards • Valid = attain measure of accuracy • Double-edged sword • Perpetuates myths regarding predictive power of models • Model validation and techniques are not context specific • Consider “business process” • Consider intended use of model & forecasts • Planning for and documentation of validation is lacking ⇒ “You get the $$$”

12. Peer Exchange SummaryImproved Validation Process Steps to verify the ability of the model system to make reasonable predictions over a range of development patterns, transportation operations, and external factors. Frank Koppelman

13. Peer Exchange SummaryImproved Validation Process • Model validation must be context specific • Type of analysis considered • Scope of analysis considered • Agency business process being supported • Model validation must consider the entire model • Consider the validity of the validation data • Document validation expectations and results

14. Peer Exchange Summary“7 Questions” • How important is it to match base-year observations? • Important, but not sufficient to establish validity • How close is “close enough?” • It all depends… • What steps are needed to ensure accurate validation data? • A conscious, planned effort • Initiate when model estimations / updates are planned • Budget necessary resources

15. Peer Exchange Summary“7 Questions” • How should model sensitivity be tested? • Forecasting & backcasting • Test range of alternatives to determine validity • Other tests to assess ability to forecast future travel? • Peer review panels • Analysis of components of change (à la FTA New Starts) • Should risk analysis of future forecasts be performed? • Depends on decisions being made…

16. Peer Exchange Summary“7 Questions” • What is the role of validation documentation in raising model credibility? • Necessary for understanding models, resulting in… • confidence in models • knowledge of the limitations of models

17. Model Validation Practices Peer Exchange White Paper • See… • http://tmip.fhwa.dot.gov/resources/clearinghouse/browse/list/6/1285 • http://tmip.fhwa.dot.gov/

18. Travel Model Validation and Reasonability Checking Manual • Draft In Preparation • Incorporates White Paper suggestions • All elements of current manual • Expanded focus beyond trip-based • Focus on temporal validation and sensitivity testing • Model validation plan specification • Identification of data requirements & budgeting • Validation documentation recommendations

19. Travel Model Validation and Reasonability Checking Manual Chapters • Introduction • Model Validation Plan Specification • Validating Model Inputs • Socioeconomic Models • Amount of Travel / Activity

20. Travel Model Validation and Reasonability Checking Manual Chapters • Trip Distribution / Destination Choice / Location Choice • Mode Choice / Auto Occupancy • Time-of-Day Choice / Time-of-Day Split / Directional Split Factors • Assignment Procedures • Temporal Validation and Sensitivity Testing • Validation Documentation

21. Update of the Travel Model Validation and Reasonability Checking Manual • Published manual • Web document • Searchable & keyword hyperlinks • Possible wiki format • Publicize updated manual • Conferences • TMIP e-mail list • Virtual Mentoring & Technical Support Center (VMTSC) • Webinars

22. Information • The Travel Model Improvement Program (TMIP) • http://tmip.fhwa.dot.gov/ • Suggestions, comments or good practices to share • Contact Sarah Sun at sarah.sun@dot.gov or (202) 493-0071