NETLIPSE Infrastructure Project Assessment Tool

1. NETLIPSEInfrastructure Project Assessment Tool Stuart Baker, Deputy Director of National Rail Projects Department for Transport, UK Vilnius, May 18, 2010

2. Contents Background IPAT IPAT – The assessment tool IPAT – The assessment process – pilot cases Lessons learned The future of IPAT Discussion

3. What is the context of IPAT? Predecessor: Netlipse I research One of the main deliverables of Netlipse II Funding by the European Commission / TEN-T Agency in order to: Improve the current management and organisation of LIPs Get insight in ‘the vitality’ of projects on certain moments, e.g. financing (gate review) Get better insight in the progress of LIPs Benchmark projects

4. What is the IPAT? The IPAT is an assessment tool The IPAT indicates the level of confidence the IPAT assessors have that the PDO and its Client have created the necessary conditions for successfully executing the next project phase The IPAT may be used by the European Commission /remark MHE/ as one of its tools for deciding what projects will receive their support.

5. When to apply the IPAT?

6. What is the added value of the IPAT? Project delivery organisations: to increase the certainty of successful execution of projects, resulting in particular in reduced cost overruns and time delays, and; Clients and funders: to understand the deliverability of projects by the project delivery organisations EU, local governments and financial institutes: to monitor and evaluate projects (ex ante and ex post) in a systematic way NETLIPSE and research institutes: the collection of information on research forecasts and future research demands.

7. Focus of the IPAT A Clear Context and Purpose, with a competent Organisation will bring the Required Results Organisation

8. IPAT – Conceptual model and themes Political context (Usually a Ministerial client) Stakeholders T6 T12 T6 T12 Related projects Private companies (contractors, advisors, operators, etc.) T 1 Political Context T 2 Objectives, Purpose and Business Case T 3 Functional Specifications T 4 Interfaces T 5 Stakeholder Management T 6 Finance T 7 Legal procedures T 8 Technology T 9 Knowledge T10 Organisation & Management T11 Contracting T12 Risks T1 Client / official sponsor T2 T3 Project Delivery Organisation T4 T5 T7 T8 T9 T10 T11

9. IPAT – model: Project Phases Phases of the Project (relate to Milestones): M 1 Initiation of the project M 2 Funding assembly M 3 Official approval official planning authority M 4 Start of execution M 5 Completion M 6 Start operation M 7 5 years after start of operation

10. Prioritisation: Level 1: Minimal importance Level 2: Little importance Level 3: Medium importance Level 4: Important Level 5: Crucial Levels of Importance by Milestone and by Theme

11. IPAT – The assessment tool Three elements: Questionnaire Scoring model Assessment process

12. A pilot assessment EX ANTE EVALUATION Femern Bælt, Denmark

13. A pilot assessment: Femern Bælt, Denmark STEP 1: Search for background information

14. A pilot assessment: Femern Bælt one of a long term series of key Danish links Øresund Great Belt Fehmern Bælt

15. 19 km coast to coast connection (Bridge or Tunnel?) Bridge Tunnel

16. Project Organisation Femern A/S: subsidiary of Sund & Bælt

17. Budget & Financing

18. Planning

19. A pilot assessment: Femern Bælt, Denmark Step 1: Search for background information Step 2: Site visit - interviews

20. Site visit Assessors team: Stuart Baker Marcel Hertogh Ellen Gehner Interviewees: Peter Lundhus – Managing director Femern A/S Steen Lykke – Project director Tunnel Open interview on 12 themes Duration: 2 x 2 hours

21. A pilot assessment: Femern Bælt, Denmark Step 1: Search for background information Step 2: Site visit – interviews Step 3: Scoring

22. IPAT- Questionnaire (Theme 1)

23. Scoring process Score all questions in the IPAT Questionnaire on a scale from 1 to 4 individually Questions that are not relevant to a project phase or not applicable to a project must be indicated as such. Note relevant arguments and comments that support the scoring as input for the assessment report Organise a meeting with the assessors team to create consensus about the scores

24. Scores • The objective of scoring the criteria is to identify the main strengths and weaknesses of the project and its organisation within each theme. • Scoring table reflects weakness of the project (low score) and strength of the project (high score):

25. IPAT – Scoring model

26. IPAT – Scoring model

27. A pilot assessment: Femern Bælt, Denmark Step 1: Search for background information Step 2: Site visit – interviews Step 3: Scoring Step 4: Analysis of the results

28. Result 1: Weight – score matrix(sub)themes

29. Result 2: Overall score total weighted percentage score

30. A pilot assessment: Femern Bælt, Denmark Step 1: Search for background information Step 2: Site visit – interviews Step 3: Scoring Step 4: Analysis of the results Step 5: Writing the assessment report

31. Two pilot assessments MONITORING / EX POST EVALUATIONS Gotthard Base Tunnel West Coast Main Line (Pre 2002)

32. Overall score: Total percentage scoreWest Coast Main Line – Failed Project Pre 2002

33. (Sub)theme scores: Weight – score matrixWest Coast Main Line – M1

34. (Sub)theme scores: Weight – score matrixWest Coast Main Line – M1, M2

35. (Sub)theme scores: Weight – score matrixWest Coast Main Line – M1, M2, M4

36. Overall score: Total percentage scoreGotthard Base Tunnel

37. (Sub)theme scores: Weight – score matrixGotthard Base Tunnel – M1

38. (Sub)theme scores: Weight – score matrixGotthard Base Tunnel – M1, M2

39. (Sub)theme scores: Weight – score matrixGotthard Base Tunnel – M1, M2, M3

40. (Sub)theme scores: Weight – score matrixGotthard Base Tunnel – M1, M2, M3, M4

41. Lessons learned from (pilot) pilots Validation and fine-tuning of the questionnaire Questionnaire was comprehensive Questions were relevant Questions were recognisable for the interviewees Questions could be answered by the interviewees Some questions are reformulated Development of the scoring methodology The assessors encountered little problems in scoring the questions The assessors were generally unanimous about the scoring The methodology of calculating the overall score was considered and changed ahead of the pilots Significance of levels of importance appears to be little Development of the final analysis of the IPAT: Still to do: what are appropriate ‘pass scores’???

42. What’s next until end 2010? Pilots still to undertake: Lock - Flanders (June 2010) French Project possibly Figueras - Perpignan (Sep 2010) Koper – Ljubljana (June 2010) Operational version of the IPAT (including assessors manual) List of competent assessors Training programme

43. Discussion (1) Role of the assessors during the assessment process: how to guarantee the quality of the assessment?

44. Discussion (2) Pass score: what is a ‘good enough’ project? (sub)theme score – position of green area??? Overall score > 50% ??? Bandwidth? Extra hurdle: the overall score might be considered good enough if on each (sub)theme level the score is in or above the green area???

45. Discussion (3) Added value of the application of IPAT: what makes the IPAT attractive to Clients and PDOs to do an assessment?

46. Time for your input now please... Having heard all this, do you broadly understand the IPAT and are you now convinced that you we can do a reliable, fair and comparable assessment of Projects with the IPAT?