1. Methodology for POLICY IMPACT ANALYSIS AND EVALUATION Paolo Roberti (Istat) Roma, RRT Meeting July 6, 2006

2. PIA*: State of the Art Discussions on PIA are constantly on the increase at the EU level: • EU “Guidelines on Impact Assessment” published on June 15, 2005; • The use of PIA methodologies is increasingly considered as a necessary condition for “good policies” (and good projects) • The investment in PIA methodologies occupy a central position in the new FP7 * Policy Impact Analysis

3. Is PIA a new fad? NO • PIA methodologies were first developed systematically in the 1950s as a tool “to tell the public what progress has been made” (US Office of Management and Budget) • PIA interests all policy areas; the outlook may be macro, meso o micro • PIA may consist of numbers, coefficients, or indicators which in turn may be quantitative or qualitative (simple or compound) factual or contrafactual and derived from complex models.

4. What does measuring the impact of policy on innovation mean ? • Measuring the impact of policy: effective and/or potential; static and/or dynamic, etc.; • Assessment of the effects and results obtained by policies: have the expected objectives been attained? Was the impact proportional to the effort and the resources employed for their implementation? etc.; • Generalization of the results: may the cost-efficient policies be replicated and implemented in other countries/regions with a “guarantee of success”?

5. PIA can mean different things. Evaluation criteria may bedemand, use-oriented o expert-driven During the assessment one must remember that • Every policy is the product of a complex and circular process that: • Continues until it is abolished; • Produces results that are not necessarily satisfactory; • Must be monitored and evaluated.

6. In the policy analysis jargon ROAMEF stands for Rationale, Objectives, Appraisal, Monitoring, Evaluationand Feedback

7. The Cycle begins with a … Problem • Which at the political level may - or may not - be considered a problem that should - or should not - be faced and resolved; • It is important to remember that even a “non policy” is a political choice; Every policy has objectives and reasons: this means results that must be reached and situations which must be acted upon

8. 1° Principle of Policy Appraisal and Evaluation “Impact and Evaluation Analysis can not be improvised ex post, it must be planned and programmed ex ante and must be directly related to the administrative and operative processes” If PIA methodologies are not planned, developed, structured and observed, the evaluation will be weak and uncertain

9. Four Policy Models Model 1: “Throwing money” on specific programmes, hoping that the expected results, general and/or specific, will somehow be reached; Model 2: Financing programmes that guarantee the attainment of precise objectives and the implementation of planned activities, hoping that the expected results will somehow be reached; Model 3: Financing programmes that entail the enactment of specific activities that are defined against specific and quantifiable objectives. Model 4: Mod (3) + activities that can be quantified + monitoring and evaluation ex-ante, in itinere ed ex-post (therefore during the entire cycle) against both specific and general, measurable objectives and results.

10. Figure 1 – Typical Programme with no defined objectives

11. Figure 4 – Typical evaluation with measurable programme objectives

12. What may be evaluated? Model 1: Respect of rules and procedures Model 2: The attainment of the established objectives: have all steps been implemented? In compliance with norms and procedures? (i.e. expense flows, costs, timing) Model 3: Did the implemented activities allow the attainment not only of the specific, but also of general objectives?

13. Model 4 The evaluation of Results may be “Good/Bad” and “Efficient/inefficient” on the basis of criteria that may require: • EVALUATION OF PROCESSES (INDICATORS) • EVALUATION OF RESULTS (INDICATORS) • EVALUATION IN TERMS OF SPECIFIC OBJECTIVES Did the innovation policies favour R&D investments? (INDICATORS & ANALYSIS) • EVALUATION IN TERMS OF GENERAL OBJECTIVES Did the innovation policies incentivise R&D investments? Did they have an impact on economic growth? (ANALYSIS & MODELS)

14. What Criteria and Which Tools may be used to perform evaluations? In one word, the answer is: Many • Facts can be evaluated with quantitative and qualitative criteria (but which facts?); • The criteria depend on the objectives, on the time and on the aspects that must be evaluated; • “Indicators” are used for rankings, scoreboards and benchmarking analysis, as well as models (simple, complex, static and dynamic), and more in general the set of tools that in the USA are referred to as “Systematic Analysis”