Transport Canada Transports Canada

1. Transport CanadaTransports Canada Program Evaluation, Transport Canada May, 2000 INTEGRATING PERFORMANCE MEASUREMENT AND POLICY/PROGRAM EVALUATION IN TRANSPORT CANADA Presentation for CES - Conference 2000 Presented by Jennifer Birch-Jones, Evaluation Manager Gail Young, Evaluation Manager Program Evaluation Transport Canada May 16, 2000

2. Objective • Share Transport Canada’s (TC’s) evolving approach to integrating performance measurement and policy and program evaluation • spheres of influence • attribution • on-going performance monitoring • research on performance challenges Page 2

3. Transport Canada’s Approach: Performance is… • Not just staying within budgets and meeting revenue targets • Not just being very busy, all the time • Not just producing a lot of high quality goods and services • Not just having satisfied “customers” • but making a difference in the real world...changing attitudes and behaviours, social and economic conditions, the state of the environment, the sustainability of natural resources, the state of the world Page 3

4. Transport Canada’s Approach: Add Value • Our evaluation unit is part of the policy shop - distinct from audit and consulting functions that reside in corporate services • We look at program and policy impacts • We have been designated as the centre of functional expertise for performance measurement, as well as evaluation • Our strategic objectives include positioning ourselves to contribute to strategic decision making Page 4

5. Transport Canada’s Approach: Work with Managers When it comes to performance, managers and evaluators both want to • Understand the policy or program logic • Keep track of what’s happening and make changes where needed • Figure out how policy/program interventions are making things happen - cause and effect • Report performance - monthly, yearly, every 5 years... Page 5

6. State Behavioral Change Behavioral Change > Changes in individuals and organizations in - awareness - acceptance - ability/capacity - action/adoption > the “what” and the “who” > direct outcomes > Changes in individuals and organizations in - awareness - acceptance - ability/capacity - action/adoption > the “what” and the “who” > direct outcomes > the vision - the ideal world > benefits to the target population, the Canadian public, the industrial sector, the resource > the “why” > long-term impacts Operational Operational Operational > internal operations > the “how” > activities & outputs > internal operations > the “how” > activities & outputs > internal operations > the “how” > activities & outputs Direct Control Direct Control Direct Control Indirect Influence Direct Influence Direct Influence Performance Story Logic - Spheres of Influence With many thanks to Steve Montague, PMN Page 6

7. Behavioral Change Behavioral Change State Target Audience > Rail Industry > General Public > NGOs > Passengers > OGDs > TSB Target Audience > Rail Industry > General Public > NGOs > Passengers > OGDs > TSB Vision > Best possible rail transportation system Safety Results> Protection of life, property and environment > Public confidence enhanced Operation Operation Operation Operation Operation > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate > Legislate / Regulate > Inspect / Enforce > Subsidize > Educate Results > Awareness > Acceptance> Ability> Adoption / Action Results > Awareness > Acceptance> Ability> Adoption / Action Direct Control Direct Control Direct Control Direct Control Direct Control Direct Influence Direct Influence Indirect Influence Rail Safety’s Performance Story Page 7

8. NaturalEnvironment Target Audience > Rail Industry > Vehicle Drivers > NGOs > Other Road Users > Provinces / Municipalities Safety Benefits > Reduction in Rail Crossing Accidents / Fatalities / Injuries > Public Confidence Enhanced Social / Economic Factors Results > Increased Road User Alertness (Awareness )> Improved Crossing Infrastructure (Ability)> Adoption of Safe Practices by Rail Industry and Road Users (Action) Direct Control Media Indirect Influence Safety Performance Story for Rail Crossings State Operation > Regulate > Educate > Incentives> Enforce Direct Influence Page 8

9. NaturalEnvironment Target Audience > Provinces / Municipalities Safety Benefits > Reduction in Rail Crossing Accidents / Fatalities / Injuries > Public Confidence Enhanced Social / Economic Factors Results > Improved Crossing Infrastructure (Ability) Direct Control Media Indirect Influence Safety Performance Story for Rail Crossing Contributions State Operation > Financial Contribution Direct Influence Page 9

10. Examples of Measures for Rail Safety Page 10 (See Attachment 1 for a more complete list)

11. Application to Policy • A similar approach can be used to tell the performance story for major policies • Can lead to some interesting observations about actual and potential government intervention to achieve policy objectives • For example…mapping the spheres of influence for government’s economic oversight of the air navigation system (ANS) raises questions about government’s role in influencing economic variables Page 11

12. TC Minister > board appts > approvals TC Minister > board appts > approvals TC Minister > board appts > approvals Sector Behaviour State of Transportation • NAV CANADA • > financially viable • > compliant with • leg/regs • - respects int’l • agreements • - respects charging • principles • - respects level of • service conditions • > responsive to users on • - price & service issues • - capital investment in • new technology • > transparent • NAV CANADA • > financially viable • > compliant with • leg/regs • - respects int’l • agreements • - respects charging • principles • - respects level of • service conditions • > responsive to users on • - price & service issues • - capital investment in • new technology • > transparent • USERS • > satisfied with ANS • price & service • > satisfied with • openness of • NAV CANADA • USERS • > satisfied with ANS • price & service • > satisfied with • openness of • NAV CANADA • > ANS proportion of air transportation costs minimized TC TC TC Monitoring ? Monitoring ? Monitoring ? TC Controls CTA user-initiated appeals CTA user-initiated appeals CTA user-initiated appeals TC Influences TC Contributes to Spheres of Influence for ANS Economic Oversight NaturalEnvironment Social / Economic Factors Media Page 12

13. Changes in variables: What happened? (PM) Success: What was our contribution to changes at the state level? (eval) Effectiveness and success: Did we make a difference? Was it the expected difference? (eval) Cost-effectiveness and alternatives: Is there a better way? (eval) Indirect Influence Direct Control Direct Influence Measurement by Sphere of Influence BEHAVIOURAL STATE Changes in variables: What happened? (PM) OPERATION efficiency (PM) productivity (PM) quality (PM) timeliness (PM) Is the intervention still appropriate? (eval) Page 13 (See Attachment 2 for more detail)

14. Why it’s Working • Shows realistic accountabilities: managers are relieved to finally see it acknowledged that they don’t control or even directly influence everything - they readily accept the logic of spheres of influence • Responds to an immediate management need to monitor, review, improve, report • Creates a meaningful link between performance management and reporting to central agencies - no longer just feeding the beast Page 14

15. Why it’s Working Continued • Helps managers do the job they know they should be doing: monitoring the performance of their own programs and policies • Lets evaluators do the job they do best • helping managers set up performance measurement systems • applying research expertise to the tough questions such as attribution • Builds a community of practice • learning • changing • sharing Page 15

16. Challenges • Objectivity, independence, co-option…need we say more? • Hiring and training the “new” evaluator - we need facilitators with people skills who can apply shared leadership in a team setting…and be solid analysts as well • Manager has to be ready and evaluator has to be flexible and adaptable - may not always get the perfect framework, measures or data • Research to determine attribution is a methodological challenge that cannot always be met, given the resources available to evaluators Page 16