EVALSED Guide

1. Evaluating Socio-Economic Development- the ever-evolvingEVALSED Guidewww.evalsed.infoElliot SternPresentation to Conference:Evaluating Public Interventions in RomaniaNovember 2nd 2006

2. EVALSED Guide Evolution of the Guide • The Means Collection 1999 developed by C3E Lyon • Guide to Socio Economic Development 2003 developed by Tavistock Institute, GHK Consulting and IRS (Instituto per la Ricerca Sociale) Milan • The Revised Guide focusing on evaluation capacity and an interactive ‘community of practice’ being developed by GHK & a network of experts

3. EVALSED Guide Philosophy of EVALSED: • Evaluation is an integral part of effective management and policy making – it is not a free standing activity nor is it just about accountability • Evaluation is not just about technique – it is also an institutionalised activity that needs to be integrated into the operations of all involved actors – which is why evaluation capacity is so important • The aim is effective socio-economic development not good evaluation on its own

4. EVALSED Guide Intended Users • Policy makers who have an interest in evaluation • Public sector managers and civil servants who commission evaluations • Programme managers who will wish to use evaluation in programmes management and planning • Programme partners involved as stakeholders • Evaluators wishing to broaden methods and strategies

5. EVALSED Guide The GUIDE is already being used widely: • The EVALSED site has over 5000 visits per month, • Each visitor clicks on about 10 pages and views about 5.5 • Italy, UK and Poland largest users Usage does not include translations: • Translations are now complete not only inRomaniabutalso in the Czech Republic, Italy and Poland • Others translations are planned

6. EVALSED Guide • Currently consists of a GUIDE targeted at non-technical readers • ‘Sourcebooks’ that are more specialist covering: • Themes and Policy Areas; • Methods and Techniques; • Capacity Building • Glossary of evaluation terms

7. EVALSED Guide • Section 1: Contribution of Evaluation to Socio-Economic Development • Section 2: Designing and Implementing Evaluation for Socio- Economic Development • Section 3: Developing Capacity for Socio-Economic Evaluation • Section 4: Choosing Methods & Techniques - Using Evidence

8. EVALSED Guide • Section 1: The benefits of evaluation; rationales and purposes for evaluation; the nature of evaluation capacity; the theories that underpin interventions; the current policy context; a review of the main types and source ideas of evaluation; the role of evaluation in the policy cycle; and how evaluation strengthens socio-economic development

9. EVALSED Guide • Section 2: The main processes for designing and implementing evaluations; involving stakeholders; defining evaluation questions and criteria; preparing terms of reference; managing evaluations; quality assurance and evaluation standards; and ensuring evaluations are useful and used.

10. EVALSED Guide • Section 3:Evaluation capacity as part of institutional capacity; the role of civil society; how to develop capacity; the architecture of evaluationsystems - centralisedand decentralised models; strengthening evaluation demand andevaluation supply including the role of evaluation societies; institutionalisingevaluations

11. EVALSED Guide • Section 4: Identifies the main factors that influence choices of methods and techniques - especially policy and stakeholder priorities and evaluation question being asked; discusses quantitative and qualitative approaches; methods for different types of programmes/ interventions; different programme stages; different evaluation stages - e.g. Scoping, obtaining information, drawing conclusions etc; how primary and secondary data is obtained; and how indicator systems are created

12. EVALSED Guide Sourcebook 1:Themesand Policies Themes: Sustainable development; increasing institutional andadministrative capacity; equal opportunities;local and territorialdevelopment; social inclusion; information society Policy areas: Agriculture; transport; employment; enterprise and innovation; justice; energy etc.

13. EVALSED Guide Sourcebook 2: Methods and Techniques Organisedunder headings: Planning and structuring, e.g. Formative evaluation, logic models, stakeholder consultation Obtaining data, e.g. Surveys, observation, focus groups Analysing data, e.g. Econometric models, expert panels, delphi techniques Evaluative judgements, e.g. Cost benefit analysis, expert panels, gender impactanalysis

14. EVALSED Guide Sourcebook 3: CapacityDevelopment National cases: Italy, Ireland, Netherlands - all nowupdated • Newcases (nearly complete):Spain, Czech RepublicandPoland • Plus a new cross cutting regional and cross border synthesis

15. EVALSED Guide Latestreview tasks • Finalise additional national cases • Review case examples – and we are accumulating practical examples of innovative evaluations, specific terms of reference, examples of evaluation use • Just finishedrevising capacity development guidance to reflect emerging experience in new member states • Introduce resources, interactivity, forum – an electronic ‘community of practice’

16. EVALSED Guide Interactivity • Overall purpose is to create a more dynamic interactive and useful site that will support a community of practice among those who evaluate socio-economic development

17. EVALSED Guide • Specific interactive elements • A resource library that can accept documents, examples and methods that will be tagged or catalogued so that others can retrieve • An online ‘forum’ or bulletin board for exchange on specific topics • FAQs (Frequently Asked Questions) to accumulate forum know-how • Occasional asynchronous ‘conferences’ on selected topics – some suggestions: indicators and data sources, participative methods of evaluation, causality and explanation

18. EVALSED Guide • The aim is to create a dynamic resource that will be an ‘evolving object’ rather than something always about to become obsolete and requiring revision! • To ensure that evaluation is both methodologically and even theoretically ‘fit for purpose’ whilst reinforcing management and policy-making systems so that they use evaluations well • But the overall aim is to support better socio-economic development that will contribute to the economic opportunities and quality of life of Europe’s citizens