Session 11: Model Calibration, Validation, and Reasonableness Checks Source: NHI course on Travel Demand Forecasting (152054A)
Terminology • Model Calibration • Model Validation • Reasonableness checks • Sensitivity checks • Special generators • Screen lines (some modelers do not think this is important) Is the model sensitive to policy options?
Key Concepts • Not enough attention on model evaluation and reasonableness checks • Checks performed after each step • reduces error propagation
Information Requirements for Validation and Reasonableness • Demographics and employment • Highway and transit networks • Base year survey • Base year traffic counts
Sources of Error • Network coding • Sampling • Computation • Specification • Data transfer • Data aggregation
Inputs and Outputs Complete? Level of Detail? Reasonable? Methodology? Source? Sensitive? Documentation of calibration? Valid for base year? Current? Reasonable?
Model Calibration and Validation Independent Data Sets Feedback Loop
Socioeconomic Data: Check Reasonableness of … • Source for estimates and forecasts • Population and household size trends (graph 1950 to present and check trend) • Household income trends (graph as far back as this goes … 1990?) • Dollar values used in forecast (use constant dollars) • If used, trend of automotive availability (S curve?) • Distribution of employment by type (basic, retail, service) over time • Employees per household and per capita … rate of increase is decreasing • Household and employment changes by zone • Area type (density) changes – are the growth areas reasonable?
Travel Survey Data Reasonableness Checks • travel survey data • Types of survey conducted • Year of survey • If no survey (borrowed) • Source of trip rates, lengths, TLFD • Is area similar • Geographic area? • pop/HH/emp characteristics? • Urban density and trans system? • Trip chaining convention • Compare to similar regions and to same region in earlier times: • Person trip rates by trip purpose • Mean trip lengths by trip purpose • HBW longest? HBO shortest? • TLFDs by trip purpose
Network Data Reasonableness Checks • Facility types, level of detail • Speed and capacity look-up table (what LOS used for capacity?) • Significant transportation projects – narrative? • Plot (facility types, #lanes, speeds, area types)
Trip Generation Reasonableness Checks • Trip production and attraction models • Form? • sensitivity? • Trip purposes used • Person trip or vehicle trip rates used? • P&A balance (0.9-1.1 ok) • Special generators (be consistent in future model)
Trip Generation CalibrationTypical Values • Person trips per household: 8.5 to 10.5 • HBW person trips per household: 1.7 to 2.3 • HBO person trips per household: 3.5 to 4.8 • NHB person trips per household: 1.7 to 2.9 • HBW trips: 18% to 27% of all trips • HBO trips: 47% to 54% of all trips • NHB trips: 22% to 31% of all trips
Trip Distribution Reasonableness Checks • Mean trip length (increasing or decreasing?) • TLFDs
1st iteration Calibrate friction factors
Mode Split Reasonableness Checks • Automobile occupancy factors by trip purpose • Basis? Constant? • Form of mode split model? • Variables included in the utility functions? • Coefficients logical? • IIA properties of LOGIT models • Mode share changes over time • Mode share comparisons with other cities
Mode Split Calibration and Validation Advanced concepts needed to formulate and calibrate mode split models • Experienced planning consultant • Calibration tasks: • Form of LOGIT model • Variables included in utility functions • Calibration of coefficients for utility function variables • Testing for IIA properties • Analysis of household survey data • Analysis of on-board transit survey data • Total highway and transit trips • Ridership by route
Trip Assignment Reasonableness Checks Compare to see where adding capacity may help • All-or-nothing assignment • Equilibrium assignment • Volume delay equation (BPR) • Screen line volumes • Time-of-day assignments • Local VMT Equil ibrium All or Nothing
Trip Assignment Calibration and Validation Assignment calibration performed last • Overall VMT or VHT check • 40 to 60 miles per day per HH in large metro areas • 30 to 40 miles per day per HH in medium metro
Other Factors Impacting Forecasted Travel Demand • Telecommuting • Flexible work hours • HB business • Aging population • Internet shopping • Roadway congestion • New modes