prof.dr. Branko Kontić, dr. Davor Kontić CIVITAS Elan Study Tour Ljubljana, 23 October 2013

1. prof.dr.Branko Kontić, dr.Davor KontićCIVITAS Elan StudyTourLjubljana, 23 October 2013 Role ofevaluation in thestageofmeasuresandprojectplanning

2. Aim of this discussion • Key message to the EC – over VANGUARD (why not POINTER?; expectingsynthesisreport) – for improving the CIVITAS strategic evaluation system (top responsibility for quality projects and efficiency of budgeting) • Potential for contributing to the benefits of CIVITAS projects at operational level (e.g., city mobility level, measures level, commonmeasures, transport policy development)

3. CIVITAS Famework

4. How does EC know that the project will meet the goals? Impact evaluation Processevaluation Integratedpackageevaluation Citylevelevaluation Cost-BenefitAnalysis

5. Evaluation Framework (by POINTER, 2009)

6. Impact Evaluation Framework (by POINTER, 2009)

7. BAU and CBA Evaluation (by POINTER, 2009)

8. ProcessEvaluationFramework(by POINTER, 2009) POINTER Support Process Topics & Issues Process Topics & Issues Form City level: 26 cities Selection Focused Measure Process Evaluation Form Measure level: Ca. 120 measures (Ca. 30%) Focussed Measure Process Evaluation Selection Measure level: 336 measures (100%) Measure Process Evaluation Measure Process Evaluation Form Preparation Implementation Operation (by POINTER, 2009)

9. Overview - LCA ofmeasuresandevaluation Phases/activities of a measure Evaluationofmeasure’s andprojectgoals planning Process evaluation implementation operation Impact evaluation CBA Transfer&upgrade

10. Missing components • Strategic view in the overall concept (not transparent and explicit; expectations to be provided by POINTER and Project Manager, however not clearly presented/formulated) • Loop, feedback, monitoring (possibility for consistent adaptation in terms of project goals – need for additional evaluation approach/understanding/tools in the planning stage of the project)

11. Missingelements in theplanning stage 1/2 • Role oftheEvaluation Plan • Purpose (holisticview); it is not only a deliverable • Project managementcontext (consultation); bettercooperationneeded • Changes in projectimplementation (measures’ design, implementation, operation) andlinks to theevaluation plan • AgreementsbetweenMLsandEvaluation Team (objectives, datacollection, etc.); seenextslide • Performanceassessment (PA) of a measure in thecontextofevaluation; seenextslide; PA followsall (internalandexternal) barriersanddrivers, processevaluationfollowsonlyoutsidebarriersanddrivers

12. Missingelements in theplanning stage 2/2 • Linksbetweenevaluationanddissemination: evaluationofdisseminationactivities, disseminationofevaluationresults • Linksandpreparations to strategicandmanagement role oftheevaluation, specifically PA and CBA: Is/wasmoneybeingreasonablyspent? Was it worthdoing? Whatcouldbe done better? Whichmistakesweshould not repeat? Post “lessonslearned” approach is lesseffective!

13. A sample of anagreement between ML and SEM

14. A sample of the Performance Assessment table

15. Possibleimprovements 1/3 • Within the hierarchy the areas of responsibility are not defined between measure, city and project level. Especially the role of the measure leader with regard to the evaluation needs to be defined clearly. • Not everyone knows how evaluation works and what is needed. The measure leaders are not experts in evaluation. • The planned evaluation procedure does not always work according to plan, it needs to be flexible. • Ljubljana SEM signed agreements with measure leaders on the collection of data, this is a good tool to clarify the role of the measure leaders, asking them to provide data on a regular basis. • There is a pressure from the PM (Project Management) regarding ticking boxes only without checking the content of the evaluation and its wider/strategic purpose. • Measure leaders need to understand what evaluation is about!! Evaluation should be continuous activity in line with measure planning and implementation.

16. Possibleimprovements 2/3 • Process evaluation is important for all measures not only focus ones; it clarifies whether implementation is progressing well or not. • Ljubljana introduced performance assessments from the early beginning. It was defined as an addition to the process evaluation – covers also internal barriers and drivers, and monitors some key components of a ML's work and performance of the measure itself (e.g. management, implementation, evaluation). The results are aggregated in a score (colour score system with comprehensive explanation of the rules for assigning a specific score/colour). • Internal Progress Report (IPR) serves primarily for management purposes; evaluation results loose their strength in the context of this reporting. Transposing performance assessment scores into "traffic light system” of the IPRs should be consistent.

17. Possibleimprovements 3/3 Role of the evaluation should be somehow integrated in the project approval procedure by the EC! Justification: In the case of ELAN more than a year after official start of the project the evaluators were working with MLs and PEM on "what is to be done in the framework of a certain measure and how will the work/measures be evaluated". The consolidation of DoW in terms of evaluation activities is necessary. This could be formalized by a requirement that a DoW quality check is done by the evaluators too, not only by the project manager and project coordinator.

18. Manager Manager Manager Manager Manager Manager Local Local Evaluation Evaluation Evaluation Dissemination Dissemination Dissemination Manager Manager Manager Manager Manager Manager Measure Measure Measure Measure Measure Measure Improvedorganizationof CIVITAS Elan European Commission European Commission European Commission European Commission European Commission European Commission Political Steering Group Political Steering Group Political Steering Group Project Coordinator Project Coordinator Project Manager Project Manager Project Manager Project Consortium Project Consortium Project Consortium Project Management Group Project Management Group Project Management Group Financial & Administrative Financial & Administrative Reporting Reporting Meeting Meeting Meeting Site Site Site Dissemination Dissemination Dissemination Evaluation Evaluation Evaluation Scientific Scientific Scientific Partners Partners Partners Coordinators Coordinators Coordinators Coordinator Coordinator Coordinator

19. Improvedorganizationof CIVITAS Elan