NATIONAL EVALUATION STRATEGY

1. NATIONAL EVALUATION STRATEGY MINISTRY OF PUBLIC FINANCE MANAGING AUTHORITY FOR COMMUNITY SUPPORT FRAMEWORK Parliament Building, Bucharest, 2 November 2006

2. Why we need to evaluate? • It provides answers to worthwhile questions • Evaluation can reduce uncertainty and improve planning and implementation BUT if evaluation is to be useful and usable, it needs to be seen as an integral part of decision making and management and indeed the entire process of democratic accountability.

3. The role of evaluation • To contribute to the design of interventions, including providing input for setting political priorities • To assist in an efficient allocation of resources • To improve the quality of intervention • To report on the achievements of the intervention • To provide lessons for future intervention • To provide answerability

4. What is (not) the evaluation? • Evaluation is a judgement of interventions according to the results, impacts and the needs they aim to satisfy. • Evaluation is not monitoring, audit or control. The evaluation’s results do not lead to repercussions on intervention’s managers or designers, but lead to changes in its design, financial allocations or in the way of implementation. It is a policy and management tool, not a tool of control.

5. Why we need a National Evaluation Strategy? • The purpose of evaluation is not well understood • Confusion exists about the concept of evaluation • Real demand for evaluation has come from outside the Romanian administration in the form of foreign donors • The institution of government is poor when it comes to using evaluation as a management tool

6. Why we need a National Evaluation Strategy? • There are already a significant number of initiatives to improve policy making and the overall quality of managing, implementing and evaluation of publicly-funded programmes. • If brought closer together, synergy could be obtained and thus the impact of the positive trends could be much more significant. • The National Evaluation Strategy (NES) is an important vehicle that could bring together all of the disparate strands currently trying to drive evaluation and will make the case stronger if stakeholders are unified and in agreement.

7. Steps involved in drafting the National Evaluation Strategy? • EC recommendation to extend to EU funded programmes evaluation practice to national funded interventions • The Single Action Plan • Review of the Romanian legal framework • Assessment of the evaluation culture in Romania

8. The Single Action Plan • Establishing the National Evaluation System among 8 specific objectives • Elaboration of a National Evaluation System and subsequent action plans • Implementation of the Action Plan for taking over the PHARE interim evaluation • Implementation of the Action Plan for establishing the National Evaluation System • Implementation of the Action Plan for raising awareness on evaluation • Implementation of the Action Plan for development of local evaluation capacity (within private market) • Approved by the Government in 2005

9. Reviewthe Romanian legal framework • Completed in February 2006 • Findings • The Law on Public Finances no. 500/2002 does not include explicit requirements to conduct evaluation exercises as part of the financial management process. It is stipulated, however, that the institutions managing public funds have the obligation to prepare “annual performance reports” that set out, for each programme, the objectives, the foreseen and the actual results obtained, indicators and associated costs. These reports are to be attached to the annual financial reports. • The Government Emergency Ordinance 45/2003 for the management of local public funds does not include an explicit requirement to conduct evaluations. When defining the term "programme", however, it states that "indicators should be established to allow the evaluation of the results obtained..", but there is no explicit requirement to carry out an evaluation of a programme or of other activities.

10. Reviewthe Romanian legal framework • Findings • The Law for Internal Audit 672/2002, regulating the audit activities for institutions that use public funds defines internal auditing as "an independent and objective activity…that helps the public entity to fulfil their objectives through a systematic and methodical approach, that evaluates and improves efficiency and effectiveness of the management system based on risk management, control and administrative processes." • The Regulatory Manual of Parliament operations do not explicitly include the right to perform evaluations of any type. It does, however, foresee the right to initiate parliamentary inquiries under certain conditions.

11. Reviewthe Romanian legal framework • Findings • The report concludes that by comparison with EU evaluation regulations, Romanian legislation has specific provisions that explicitly require evaluation only for the co-financing budgets for EU and other donor funding and for research programmes. For the other types of activities, even though mention is made of it, requirements to conduct evaluations are either not clear enough or not enforced properly.

12. Reviewthe Romanian legal framework • Recommendations • Romanian legislation related to evaluation needs to be upgraded in line with EU legislation and should address all of the areas and institutions that are managing public funds, as a management tool for public administration. New legislation should be supported by methodological guidelines that explain how to apply the law. • Moving from the current annual budgeting cycle to a multi-annual budget programming cycle would nurture the development of the practice of evaluation. Furthermore, evaluation would be easier to conduct if budgets were allocated on the basis of identified programmes and projects, as opposed to an allocation by type of expenditure only.

13. Reviewthe Romanian legal framework • Recommendations • The institutional capacity of public administration to manage programmes and projects should be improved • There should be a clear distinction between monitoring, audit, control and evaluation.

14. Assessment of the evaluation culture • Conclusions • Evaluation has been a constantly developing practice for donor-funded programmes, but is still under-developed for the activities funded by the Romanian national budget. However, there is clear evidence of growing interest and evidence of actions taken at various levels to improve the management of public policies and programmes, including evaluation. • The legal base for evaluation is gradually improving. It should be noted that equally important to enacting new legislation is the enforcement process, because past experience in Romania suggests that even when certain legal provisions existed, they were either not well understood or simply not enforced.

15. Assessment of the evaluation culture • Conclusions • Three main areas of further action can be identified: • Further development of the legal framework; • Preparing an adequate know-how base: methodologies, guidelines, publications. • Increasing the capacity of the public managers to manage the evaluation process • On the first two areas there are already a significant number of initiatives to improve policy making and the overall quality of managing, implementing and evaluation of publicly-funded programmes. If brought closer together, synergy could be obtained and thus the impact of the positive trends could be much more significant. It is encouraging to notice that already this process of co-operation between the various initiatives has started and it is recommended that this is maintained and increased.

16. Assessment of the evaluation culture • Conclusions • The National Evaluation Strategy (NES) is an important vehicle that could bring together all of the disparate strands currently trying to drive evaluation and will make the case stronger if stakeholders are unified and in agreement. It is to Romania’s credit that it is taking this step and, given the current efforts, there is a great opportunity now to be taken in developing an evaluation culture which will underpin more effective governance within the country. • Co-operation between the structures in charge of managing evaluation for the NSRF process and the structures that will manage the evaluation of nationally-funded activities is important for at least two reasons: a) the experience of evaluating the NSRF OPs will be a very valuable source of lessons; b) both structures should aim to establish a unified approach to evaluation, applied for European funds as well as for the national funds. To this end, it is important to clarify how evaluation units in MA’s and line Ministry PPU’s will work together.

17. Assessment of the evaluation culture • Conclusions • In terms of managing the evaluation process, it is the view of the authors that evaluation is an integral part of management and as such it should be embedded in the everyday practice of public managers. It is the public managers themselves who must either carry out internal self-evaluations or commission them to external contractors. This is the sole way to promote evaluation as a managerial tool. • The other use of evaluation is that of control, for accountability reasons. This requires the commissioning of evaluation exercises by someone other than the entities under evaluation and justifies the existence of independent entities in charge of managing evaluation processes, such as the Court of Accounts or independent units created within the public administration apparatus.

18. Assessment of the evaluation culture • Conclusions • Moving from a legalistic framework to a performance management framework is a necessary step if evaluations are to move away from mere reporting and become a tool for improvement. Experience suggests that such a process may take up a considerable amount of time and it is recommended that the managers of the National Evaluation Strategy implementation bear this in mind and are prepared for a long-term sustained effort.

19. Assessment of the evaluation culture • Recommendations • The National Evaluation Strategy should avoid over-ambitious objectives, particularly in the short to medium term. Experience in other EU Member States and as documented in the Evalsed Guide has shown that a slow and steady burn has proved most effective in building an evaluation culture in other Member States. • Create stronger links and maintain regular communication between the main actors who are promoting the new practices of evaluation • If possible, agree that one of these actors takes a lead role in the implementation of the NES and the overall promotion of the practice of evaluation in the country, while each actor can retain leadership in their respective areas.

20. Assessment of the evaluation culture • Recommendations • Establish first, between these actors, a common understanding of the concepts related to evaluation, of its scope for application and of its vocabulary. The constant interchanging of terms and titles is too confusing and if people at the top are confused, there is no doubt this confusion will translate through the system. • Attempt to harmonise the methodologies and manuals that are currently being prepared by the various actors. • Once the NES has been agreed and approved, divide the tasks such that each of the actors has clear responsibilities for a certain set of actions.

21. Assessment of the evaluation culture • Recommendations • Each actor should conduct a few pilot evaluations in their respective areas and then should cross – reference the results between the actors. This could involve institutions which are most open to the idea so that a model of good practice is established at an early stage, that people can see how it is done and how it is used to dispel any fears they might have. This will assist in trying to move from a culture of control and reporting to a culture of learning and analysing. The pilot evaluations should be followed by a number of subsequent exercise aiming to improve the methodologies applied until a satisfactory process is obtained, that can be transposed into a model and a methodology manual.

22. Assessment of the evaluation culture • Recommendations • It is important to ensure that all Ministries have some degree of ownership of the National Evaluation Strategy, and this means intensive consultation and involvement of heads of departments while preparing and rolling it out. • At the same time as demand is being stimulated, measures to increase supply and capacity should be put in place, concentrating on inside and outside the public administration system.

23. Principles ofthe National Evaluation Strategy • Co-operation • close working between the structures in charge of managing evaluation of the ongoing, pre-accession instruments, the NSRF process and the structures that will manage the evaluation of nationally-funded activities is important for two reasons: a) the experience of evaluating the NSRF OPs will be a very valuable source of lessons; b) both structures should aim to establish a unified approach to evaluation, applied for European funds as well as for the national funds. To this end, the evaluation units in MA’s and line Ministry PPU’s should participate in the evaluation networking activities.

24. Principles ofthe National Evaluation Strategy • Co-operation • Cooperation between the Ministry of Public Finance MACSF ECU and the Ministry of Administration and Interior’s MA for the Administrative Capacity Development OP is vital, as implementation of the activity planned under the evaluation Key Area of Intervention in the Administrative Capacity Development OP will form a central part of implementation of the Strategy.

25. Principles ofthe National Evaluation Strategy • Ownership • this Strategy can only be implemented if there is clear ownership of and commitment to the Strategy. Again this will necessitate ongoing cooperation between the MACSF in the Ministry of Public Finance and the MA for the Administrative Capacity Development OP. The Strategy has a greater chance of being implemented if other line Ministries and the wider public administration also feel that they stand to benefit from its implementation. It is important to ensure that all Ministries and local authorities have some degree of ownership of the National Evaluation Strategy, and this means intensive consultation and involvement of heads of departments while preparing and rolling it out.

26. Principles ofthe National Evaluation Strategy • Independence • Evaluations should be managed by someone other than the entities under evaluation and this explains the existence of independent entities in charge of managing evaluation processes, such as the independent units created within the NSRF OP Managing Authorities and the Public Policy Units. • Long-term vision • Moving to a performance management culture involves understanding evaluations as a move away from mere reporting towards becoming a tool for improvement. Experience suggests that this may take up a considerable amount of time and it is recommended that the managers of the National Evaluation Strategy implementation bear this in mind and are prepared for a long-term sustained effort.

27. Principles ofthe National Evaluation Strategy • Independence • Evaluations should be managed by someone other than the entities under evaluation and this explains the existence of independent entities in charge of managing evaluation processes, such as the independent units created within the NSRF OP Managing Authorities and the Public Policy Units. • Long-term vision • Moving to a performance management culture involves understanding evaluations as a move away from mere reporting towards becoming a tool for improvement. Experience suggests that this may take up a considerable amount of time and it is recommended that the managers of the National Evaluation Strategy implementation bear this in mind and are prepared for a long-term sustained effort.

28. Principles ofthe National Evaluation Strategy • Developmental • It is important to move in a planned and staged way from a culture of control to a culture of learning and analysis. Institutions that are most open to this idea could conduct some pilot evaluations (under the proposed “evidence-based management” field of intervention), so that a model of good practice is established at an early stage. People could thus see how evaluation as a means of learning is done and which may dispel any reservations they might have. Pilot evaluations could be followed by a number of subsequent exercises aimed at improving the methodologies applied until a satisfactory process is obtained, that can be transposed into a model and a methodology manual. • Evaluation is an integral part of management and as such it should be embedded in the everyday practice of public managers. It is the public managers themselves who must either commission evaluations from external contractors, or, when sufficient internal expertise exists, may even initiate self evaluations. Taking a direct interest in the design and use of evaluation is essential in order to promote evaluation as a managerial tool

29. Principles ofthe National Evaluation Strategy • Transparency • this Strategy has been drafted following a lengthy consultation process with key stakeholders in the Romanian public administration, including officials in the Ministries of Public Finance, European Integration and Administration and Interior, the General Secretariat of the Government, the Romanian Parliament, the Delegation of the European Commission and members of Romanian academia, civil society and the policy research community. As such, this Strategy expresses the divergent views on the necessary future direction of evaluation of many public and non-public stakeholders and is not the “voice” of either the MACSF in the MPF, or the MA for the Administrative Capacity Development OP.

30. Principles ofthe National Evaluation Strategy • Accountability • The Strategy assigns implementation tasks to various public bodies. It is essential, in order for the Strategy to be implemented, that these public bodies are answerable for the implementation of the tasks assigned to them. It is vital that both the MACSF in the MPF and the Administrative Capacity Development OP Managing Authority engage other line Ministries and local authorities to take ownership of the Strategy’s recommendations and action plan.

31. General Objective ofthe National Evaluation Strategy • The overall aim of the National Evaluation Strategy is that “there will be a functioning national evaluation system, the parts of which reinforce each other; encompassing the public and private sector and civil society; and contributing to the effective management of public interventions and the accountability of policy makers and public managers. There shall be recognition of the importance of evidence based policy making.”

32. Operational Objectives ofthe National Evaluation Strategy • Evaluation training and HRD resources are available from various sources (civil service, private training providers, universities) and open to participants from within and without the public sector – starting with 2007 • Professional networks are established to provide a medium for the exchange of ideas/good practice, professional development, the development of standards, etc. They will enjoy the participation of civil society as well as public service participants – starting with 2007

33. Operational Objectives ofthe National Evaluation Strategy • A central government ‘resource’ will be created as a repository of evaluation expertise. It will promote good practice by networking across all departments – starting with 2007 • There is capacity at all levels of government to commission or carry-out evaluations, and to know the right evaluation questions to ask – 2010 for Central Public Administration; 2011 for Local Public Administration

34. Operational Objectives ofthe National Evaluation Strategy • Quality and appropriate monitoring systems are in place at all levels of government (local and central) and in any agency where public money is spent – 2011 for Central Public Administration; 2012 for Local Public Administration • Evaluation is integrated into the procedures for the formulation, delivery and follow-up of all public interventions, regardless of funding lines. Documented procedures will ensure diligent follow-up of evaluation results/recommendations- 2013

35. Funding sources ofthe National Evaluation Strategy • PHARE 2005 – 4 million Euro • NSRF and Rural Development and Fisheries OPs Technical Assistance priority axes– approximately 13 million Euro • Key area of intervention of the Technical Assistance Operational Programme of NSRF – 8 million Euro • Key area of intervention of Administrative Capacity Development OP – 30 million Euro • TOTAL approximately 55 million Euro

36. Fields of intervention of the National Evaluation Strategy • Effective evaluation of the EU funded programmes • Evaluation training for civil servants, private, academic and civil society • Resource and network support • Evaluation Facility aiming at encouraging Public Administration to commission evaluations • Obtaining evaluation relevant data • Embedding of evaluation in public interventions design and management through a normative act