The Evaluation of CSF 2000-2006 The Rural Development Evaluation Experience

1. The Evaluation of CSF 2000-2006The Rural Development Evaluation Experience Sabrina Lucatelli, UVAL sabrina.lucatelli@tesoro.it

2. A Brief Descriptionof Rural Development Policy and the way it works in Italy Mid Term Evaluation: Some Information on the content The Steering Group and the Process The Updating of the Evaluation The Wider UVAL Activity on Rural Areas

3. Regions Objective I Regional Rural Development Plans (European Agricultural Guarantee Fund) Operative Programmes (European Agricultural Guidance Fund+ Structural Funds) Other Regions Regional Rural Development Plans (European Agricultural Guarantee Fund) In all regions there is Leader + Programme Rural Development and Programming OrganizationPolicy Based on Council Regulation 1257/99 No Central Rural Development ProgrammeBut Regional Rural Development Plans

4. For Objective I Regions • CAP accompanying measures are included in RRDP (early-retirement schemes; agri-environment measures; forestry; less favoured rural areas) • All Other Measures are included in the Operative Programmes • They are included in parts of the Programme concerning: Natural Resources and Local Development • More than 60% of total rural development funds are allocated within operative programmes (together with other interventions)

5. Operative Programmes and Rural Development(Ojective 1 Regions) • Sectoral Perspective and Area based perspective • Two main objectives (from the CSF): Improvement of agricultural systems and agro-food chain competitiveness Development of Rural Territories throught the valorisation of local resources (agriculture and non agriculture ones) • Rural Development as a horizontal evaluation priority

6. Some Financial Information FEOGA Allocations within the CSF 2000-2006 Public Expenditure (Allocations) • 5.057 Millions of Euro (FEOGA) • 41.444 Millions of Euros (CSF) • 12 % of total CSF allocation is covered by FEOGA • 29,5 % of Axes 4 (Local Development) is FEOGA Funded

7. What the Commission asked from EvaluationDG Regio Guidelines and DG Agri Guidelines DG Agri Preoccupations: Rural Development Policy as the Second Pillar of the CAP; The Coherence of the Rural Development Policy (one Regulation/one Policy) Common Evaluation Questions, Criteria and Indicators Comparability between Different Countries

8. What National Evaluation System suggested to Evaluators and Managing Authorities for the best implementation of Rural Development Evaluation Activity • Definition of an Evaluation Design • Evaluation of strategy’s validity • Coherence of intervantion adopted and selection criteria • Evaluation of results and impacts: efficience and effectiveness • Evaluation of management system (including partnership and institutional improvements) • Evaluation of monitoring systems

9. Mid Term Evaluation – Evaluation Approach and Main Results December 2003 • A General Assessment of the Programme • Measure Approach, mainly with a financial perspective (eg. Total Cost; Total Allocation; Total Spent ) • “Reading” measures and interventions through a special reclassification of them Reflecting the Strategy of Rural Development and the Main Objectives

10. Objectives and Grouping of Measures • Competitiveness Improvement • Agricultural Farms Modernisation • Services in favour of Agricultural Farms • Infrastructures in favour of competitiveness • Development of Rural Areas • Modernisation of Rural Areas • Rural Infrastructures • Services in favour of Rural Areas • Activity and Income Diversifications • Environment • Environmental Protection • “Environmental Modernisation” • Environmental Services • Environmental-Forestal Infrastructures

11. Main ResultsTotal Contribution per Objectives (31-12-2003)Similar Situation taking into account Allocations and Total Spent

12. A consistent imbalance of programmes in favour of the Competitiveness Objectives • Almost all Regions were privileging the Competitiveness Objectives; • Within this objective, 87% for Farm Modernisation (and just 4% in services in favour of farms); • Little investment in favour of Development of Rural Territories, mainly infrastructures (Only Campania 27% of total contribution); • Little investment in favour of Diversification of rural Economy (4% of the total); • Very little investment in Services to improve quality of life of people living in rural areas

13. Quite Good General Assessment of the Programme, but very little Evaluation of Results and Impacts • Measure Approach, mainly with a financial perspective (eg. Total Cost; Total Allocation; Total Spent ) • Rarely the measure analysis is based on a true evaluation question; • In analysing measures results, no use of territorial analysis; • Very rare use of physical monitoring figures; • Very little results and impacts analysis • NOT 100% satisflied by the Quality of the Work!

14. The Process and The Steering Group Activity • Continuity; • Accompanying and Following the Research; • Heterogeneity of the Steering Groups Components; • A smooth group (Quick Decisions; Quite High Technical Expertise) • Quite informal and direct relationships • Sometimes, quite high level of discussion and also disagreement

15. The organisation of a “public event” to diffuse rural development Evaluation Results • A Workshop to discuss Evaluation Results with: • Rural Development Public Officers • With Single Programmes Evaluators • With Social Partners and Local Operators … The idea was to discuss on “True Rural Development Issues” with people interested on the subject, even if extern to The Programming Circle … A way to unblock the situation …

16. Why end up deciding to organise a workshop? • Workshop as a tool of dissemination, but also as an important step of the Evaluation Process • A way of drive out the Evaluator • An occasion to open the discussion between Evaluators; Programme managers and Subjects interested by the programmes (Beneficiaries; Producers Associations; Local Actors) • An opportunity to clarify Evaluation Messages • To give Programme Managers the occasion to react to Evaluation judgements • To give Evaluators an occasion to open a methodological discussion between themselves

17. Results of the workshop • Very High Participation (82% of total invited) • Confrontation and returns/suggestions to improve evaluator’s Report • Methodological suggestions: integrated measures approach; territorial approach; more focused and evaluation’s questions driven approach • Suggestions to improve evaluation process (the idea to implement ex-post evaluations) • Training opportunity for evaluators and especially for Regional Managing Authorities

18. The Updating Evaluation Phase • Improving Evaluation Questions Quality; • Needed a stronger involvement of social partners and people concerned by the Programmes; • A stronger attention to effective Results and Impacts: Are People stopping to leave rural areas? Is the quality of life improving in rural areas? • The object of Evaluations are Rural Territories and people living in rural areas: not just farmers! • To use more infiled research methods • Need to improve Ingrated Approach (between measures but also between different programmes working in the same area);

19. Selected Evaluation Questions for Rural Development In general, to concentrate on best practices of programme implementation and using also territorial approaches • To taste how programmes has been effective in Improving Quality of the Agro-Food System in Southern Italy • Effectiveness of Interventions in favour of Rural Territories Development and Improvement of Quality of life in Rural Areas • Relationships between Effectiveness of Rural Development Programmes and the existing Governance options • Services Supply, infrastructures and Quality of Life in Rural Areas (is there integration between different programmes’ tools?)

20. Case-Study ApproachWhy? • Need to move from a general knowledge of the programme mainly based on documents and monitoring data, to a more focused and field research methods; • Necessity to use Primary Data and to turn to Questionnaires and Focus Groups • To recover a relationship with beneficiaries and a new Knowledge of Territories Results coming from each case studies have been used to answer to all evaluation questions

21. Two Case Studies with a Food Chain ApproachThe Pecorino Romano chain in SardegnaThe wine industry in Sicily Two Case Studies with a Territorial Approach The Fortore area in Beneventano Province The Integrated Programme (PIT) in Alto Basento (Basilicata)

22. Main Findings • Case Studies showed how different Funds are simultaneously working on specific territories, but without coordination and without a “core strategy” • Integrated Programming tools such as PIT are only partially helping this process to take place • Different Governance models with different pro and contra (The Fortore Case an example of good decentralisation with scarce relationships with local actors; The PIT Alto Basento case a better local partnerships, but a lack of decisional power) • A general under evaluation of the problem of the quality of life in rural areas (little financial investments but also problems of new services management – ordinary policy problems)

23. For more information on the Workshop see • Materiali UVAL n. 7, “Evaluation for Development of Rural Areas: an integrated approach in the evaluation of development policies” http://www.dps.tesoro.it/materialiuval/documenti.asp • See also website of OECD, the work of Working Party on Territorial Policy in Rural Areas GOV Department www.oecd.org