Participatory Evaluation 7th May 2010 Budapest

1. ex-post and on-going evaluations of eu development programmes by the European commission and the visegrad countries Participatory Evaluation 7th May 2010 Budapest

2. Agenda • DTI and our evaluation experiences • Participatory evaluation in general • Case - Vocational Education and Training system • Case - VET reform 2000 ex post evaluation • Lessons learned

3. Evaluation assignments • Leonardo programme evaluation • Observing change in VET – Leonardo II programme • Impact assessments and ex ante evaluations in the area of audiovisual and media policies (DG information Society) • Impact assessments and ex ante evaluations in the area of ICT deployment, innovation, public services and quality of life (DG information Society) • Cedefop project evaluations (DG Education & Culture) • Mid-term evaluation of LEADER+ in Denmark • Thematic ex post evaluation of Structural Fund support to enterprise development and innovation (DG Regio) • WING - study for the impact analysis of FP5 eGovernment, e-learning and e-business (DG information Society) • + many evaluations at national level in Denmark, Sweden and Norway.

4. Overview of participatory methods UNDERLYING PRINCIPLES • embracing complexity • recognition of multiple realities • prioritising the realities of the disadvantaged • grassroots empowerment • from assessment to sustainable learning • relating learning to action TECHNIQUES: • Visual techniques (diagrams, ranking, mapping, etc) • Consensus conferences • Group and team dynamics methods (incl. Role play) • Photo and video productions & diaries • Case studies, semi structured interviewing, participant observation, etc.

5. Participatory Evaluation CASE 1: Evaluation of VET system

6. Political frame Skilling – theory, practice and assessment Institutional frame Organisational frame Activity Complex social VET system Colleges and company National bodies, qualification structure, courses, assessment bodies, inspection Law, ministerial notes, regulations

7. Policy - the broader the scope – the more complex

8. Changing and evaluating social systems • Changing and evaluating - directed simultaneously at all levels • Interactivity and consistency between the different layers are the main requirements for effective change • It is a continuous process of incremental development sometimes combined with ”earthquakes” • Social systems are embedded (for example VET sits between economy, education and social affairs • Reasons for change can be endogenous and exogenous • Multi layer, Multi actor and multi purpose character!

9. ”I can’t change the direction of the wind- but I can set my sails so I am sure to reach my destination” • The notion of educational planning: implicitly suggest clearly defined objectives, mutually exclusive choices, undisputed causal relationships, predictable rationalities, and rational decision-makers. • BUT IS THAT THE REALITY • Increasing complexity- devolution of governance • Educational planning- overlapping episodes, multiple actors and divergent perspectives. It entails the processes through which issues are analysed and policies are generated, implemented, assessed and redesigned.

10. A fluid perspective on policy evaluation • (i) Analysis of the existing situation. • (ii) The generation of policy options (participatory). • (iii) Evaluation of policy options (participatory). • (iv) Making the policy decision. • (v) Planning of policy implementation (participatory). • (vi) Policy impact assessment (participatory). • (vii) Subsequent policy cycles (part...).

11. Participatory Evaluation CASE 2: Ex-post evaluation of VET policy reform.

12. Ex post evaluation and participatory methods • Evaluation of VET policy reform 2000 • Focus on implementation of reform • … and key challenges observed Organisation Evaluation and quality Competences Theme Pedagogical toolbox Inter play

13. Themes covered by VET Reform evaluation • Teacher teams and new organisation types (teams, subject groups, mentors, etc.) • Contact teacher scheme (contact teacher for each pupil) • Modulerisation and ”whole picture” focus • Pedagogy and learning styles • School collaboration (between schools – pupil plans, education offer, teacher collaboration, etc.)

14. Participatory methods used • Self-evaluation of collaborative efforts between different actors • Explorative workshops to assess learning points from pilot activities across institutions • Reflective assessment of scaling possibilities in multi-actor workshops

15. Results • Recommendations for scaling-up activities • Practical idea catalogue for dissemination and further implementation at all schools • Greater awareness among the actors in relation to their role, strategies for the continued implementation of the reform

16. Learning points • Evaluation as support for policy implementation • Governance of innovation • Both Top-down and exploiting grass root initiative • Systematic use of ”pilots” • Participatory methods require a need for further action (almost always the case)

17. Lessons learned from participatory evaluation • Take into account the political context and timing (e.g. Elections, economic crisis) • Aim for a truly inter-sectoral process (where relevant) • Conducting a participatory evaluation takes time and resources, but it is worthwhile! • Reach consensus on the definition of key concepts • Address concerns related to participatory processes • Institutional context and individual competences and interests • Strong leadership is central to the sustainability of the evaluation initiative • Conducting participatory evaluation in itself can be an empowering process

18. Consensusconferences

19. Purpose of ConsensusConferences • Enrich and expand the scope of traditional debate between experts, politicians and interested parties by communicating citizens’ views and attitudes on potentially controversial technologies. • Citizens’ views play an important part in society’s overall technological assessment, as many technologies have direct bearing on the daily lives of ordinary people. • Qualifies people’s attitudes - they are given all the information they require until they are ready to assess a given technology. • Citizens’ thoughts and recommendations come face to face with the arguments and attitudes of politicians and interested parties.

20. Planninggroup (of experts) The tasks of this group include: • Preparing guidelines and ensuring wide accessibility of the introductory material for the citizens’ panel • Commenting on and approving the introductory material • Approving the composition of the citizens’ panel • Contributing in the selection of experts to the expert conference panel • Commenting on and approving the conference programme

21. Selectioncriteria for the Citizens Panel and tasksallocated Selection • Must be representative • Open minded and interested in debating the issues in question • Represent a broad experience base in relation to the conference topic Tasks • put qualified questions to the expert panel and on the basis of their answers formulate the final document (conclusions and recommendations to politicians). • In addition to the 4-day conference, panel members must commit themselves to participating in 2 preparatory weekend sessions. • Initially reading an introductory paper (written by journalist) • Work with the process consultant managing the process to express attitudes and messages within the panel

22. Suitability of consensusconferences Consensus conferences are suitable in connection with: • A topic of current social relevance • Regulation requiring public support The method is also suitable when: • There is a need for further public awareness and debate • There is a need to identify attitudes and objectives • There is a special need for public input • The policy-makers have to be able and willing to listen and take the results seriously as proposals from the public.

23. Exampleswithin the framework of the Danish Board of Technology (DBT) A few examples of consensus conferences held by DBT: • How can we assign value to the environment? (2003) • Testing our genes (2002) • Road pricing (2001) • Electronic surveillance (2000) • Noise and Technology (2000) • Genetically modified food (1999) • Tele-work – near and far (1997) • The Consumption and Environment of the Future (1996) • The Future of Fishing (1996) • Gene Therapy (1995)

24. Important future perspectives • Development of scenario workshops so that citizens can be involved earlier, at the stage of design and selection of criteria for developing technology. • Strengthening the power of consensus conferences to present the broad approach of citizen panels as just as legitimate a criterion in technology decision-making as the more narrow approach of experts. • Both methods have demonstrated great capacity for creating awareness of methodological innovations and diversification, and for building networks.

25. ParticipatoryEvaluation Case 3 – Leader+ evaluation – sustainable regional development

26. Participatoryevaluations • LEADER+ • Goodgovernance • Participation at all levels • Sustainability • Gendermainstreaming • Transparency • ”Multi-level” governance (EU, national, regional) • Focus: On-going policy processesratherthanresults • The aim: To develop an innovative processorientedevaluationmethod for sustainable rural development by includinglocalknowledge

27. Success factors assessed through ex-post perspective • Pressure of problems • Project design that supports win-win coalitions • Short term success • Manageable structures and opportunities for linkages • Powerful interceders and partners • Learning aptitude and exchange of information • Transparency, process competence and flexibility • Participation • Regional promoters as policy entrepreneurs • Critical mass of labour time and money for the regional management • Competent regional management

28. The evaluationcycle • The ongoing evaluation of the political processes at the regional level on the basis of the success factors (previous slide) • Empirical research of the characteristics of the success factors at the regional level. Identification of the requirements for policy advice • Delivery of intermediary results of the evaluation to the partners in the project and possible the expert or external consultant • On the basis of intermediary results, the consultant gives advice to the regional stakeholders in order to develop a method of regional management and self-evaluation. • If the intermediary results show important problems in a region, the consultant intervenes into the ongoing process with the intention of changing the behaviour of the actors • Evaluation of the impact of the policy advice on the policy process

29. Keychallenges • Difficulties in measuring the value of thesequalitativeindicators to evaluate the politicalprocess. • Solution: • Anonymity • Severalqualitativesources • Examination of presscoveragecombinedwiththeoreticalevaluationmethod • Not all regional stakeholders and promotersarewilling to take the advice of the consultant • Hence the method is bestused in regions were the stakeholders and promoters have lessexperiencewithsustainable regional development.