Presentation 3:

1. Presentation 3: “Practical experiences gained from carrying out case studies in relation to different policy themes and/or overall context settings” Dr. Thomas Stumm Managing Director of EureConsult S.A. (Echternach, G.D. of Luxembourg)

2. Evaluating Structural Funds interventions in one Member State:Telling the story from a country-specific perspective My experience confirmed to me that in-depth case studies have great potentials for unveiling a number of issues which are normally not perceivable at a first glance by any horizontal evaluation.  The precise but often also diverse nature of impacts resulting from a policy intervention in a given region (e.g. “What has been achieved in terms of direct/indirect effects and intended/unintended effects?).  The way in which a policy intervention is implemented and working in practice in the particular region being examined (e.g. “How has a policy intervention achieved the impact?).  The influence of contextual factors or overall developments in order to locate the effect of a particular intervention or the way how a policy intervention has been realised within these (“Why” is the “what” and “how” of a policy intervention like it is?).

3. Evaluating Structural Funds interventions in one Member State:Telling the story from a country-specific perspective If multi-site case studies are realised for a specific country, an appropriate presentation format needs to be developed when elaborating a “general framework” for carrying out the case study work ( core teams of the main contractor). It has proved very useful presenting the overall outcome of multi-site case studies realised for a single Member State in a wider format (e.g. in form of a “country report”).  General context analysis preceding the actual case study part.  Subsequent in-depth analysis of the cases under review.  Final synthesis relating in-depth analysesto initial working hypotheses / the general patterns of the country context.

4. Evaluating Structural Funds interventions in one Member State:Telling the story from a country-specific perspective A sound initial briefing of the field teams carrying out case study work at a later stage should be realised,e.g. in form of a one-day “kick-off workshop” bringing together all parties involved in an evaluation. The field teams / local experts need to get a clear understanding about the wider purpose and strategic objectives of the overall evaluation ( task of the client). The field teams / local experts must be made fully aware of the central purpose of the case study within the wider evaluation ( task of the core team). The field teams / local experts must become fully familiar with all aspects relating to the practical realisation of case studies ( task of the core team). Due to my positive experiences made with such briefings, I recommend this to be a mandatory first step in any evaluation involving extensive case study work.

5. Evaluating Structural Funds interventions in one Member State:Telling the story from a country-specific perspective Field evaluators should realise a case study-based in-depth analysis with a certain degree of “distance” and “flexibility”.  As the preparation / elaboration of a good case study needs time, try to take some distance with respect to the inevitably occurring time-pressure & keep up professional standards.  Be flexible in the actual preparation & realisation of stakeholder interviews (e.g. drop questions or add new ones; carry out single-person interviews and structured group discussions)  When drawing up the case study report, try taking some distance with respect to an initially defined reporting template. Intuitively “co-determine” the importance given to a pre-defined item on ground of your actual knowledge & with respect to the actual relevance of the topic within the wider story to be told about a case.  Show intuitive flexibility while elaborating an informative and analytical but still appealing narrative which tells the story about a specific case (e.g. by focussing the story on a number of “critical incidents”).

6. Evaluating territorial co-operation programmes & projects: Telling the story from a multi-country perspective The use of the case study method is particularly recommended for evaluating European Territorial Co-operation (ETC). This is due to the fact that the complexity of real-life interactions within co-operation processes is considerably higher if compared to those taking place in interventions carried out in a single-country perspective. Case studies can - in principle - be used at three different levels: (1) When evaluating Community-level funding schemes such as the “old” INTERREG Community Initiative or the new ETC-Objective. (2) When evaluating an individual cross-border, transnational or inter-regional co-operation programme. (3) When evaluating a specific co-operation project.

7. Evaluating territorial co-operation programmes & projects: Telling the story from a multi-country perspective According to my practical experience, however, case studies on ETC are also a very demanding and particularly challenging task for each evaluator involved.  The evaluator has to be familiar with more than only one country-specific context (e.g. basic political-administrative settings, different legal & regulatory systems, different cultures & ways of “doing” and “thinking”).  The evaluator needs to have well-developed and appropriate language capacities (esp. in the context of cross-border co-operation).  The evaluator needs to have a well developed understanding about the previous history of co-operation (i.e. long-term development & core activities carried out in the past).  The the activities for preparing and realising an ETC-case study are much more complex as a greater number of key stakeholders originating from different countries and from different levels of co-operation need to be identified / contacted for telling a story which is comprehensive and well-balanced.

8. Evaluating territorial co-operation programmes & projects: Telling the story from a multi-country perspective (1) An evaluation of Community-level funding schemes Compared to the previous INTERREG II ex-post evaluation, the current ex-post evaluation of the INTERREG III Community Initiative (2000-2006) has adopted a very different approach: Case studies play an important role in the entire evaluation process.  Based upon an extensive and aggregated analysis realised previously across all programmes, a typology of INTERREG III programmes was elaborated on ground of which a case-study sample was selected which well-reflected the diversity and types of existing programmes.  These 16 programme case studies, with each of them also involving the realisation of 5 shorter project-case studies, have shed more light on specific aspects which have already been analysed at a more aggregated level across all programmes.

9. Evaluating territorial co-operation programmes & projects: Telling the story from a multi-country perspective (2) An evaluation of individual ETC-programmes Programme-level evaluations (e.g. the ongoing evaluations for the 2007-2013 period) should make more use of the case study method especially when it comes to assessing their project-level implementation.  This can provide the strategic “programme owners” with a much better understanding about the complex reality of the project-based implementation of their programme and about the real scope and nature of the effects generated by the approved projects.  The outcome of such project-level case studies is also of a long-term relevance, as it helps building up a permanent & precious “historical memory” about the client dimension of a programme. Another interesting perspective is the use of case studies in the context of future ex-ante evaluations for ETC-programmes after 2013.  Case studies can relate to various aspects in the process (i.e. problem assessment & strategy building and design of the programme-wide decision-making/management and implementations structures).  Case studies can adopt both a programme-internal focus (e.g. Learning from the past for the future) or a programme-external focus programme (e.g. Learning from others how they have dealt with a certain issue?).

10. Evaluating territorial co-operation programmes & projects: Telling the story from a multi-country perspective (3) An evaluation of individual ETC-projects I am strongly convinced that the use of the case study method can also make an important contribution to raise the overall quality & depth of an individual project evaluation (if those are realised).  Project-focused evaluations are not yet realised very frequently: Reasons might be a not yet very broadly developed awareness about the associated benefits & the still rather weakly developed methods or techniques for evaluating ETC-projects.  An external evaluation should be carried out if a project has (1) a certain budget size (e.g. > € 2 million) and/or (2) if it has a rather complex implementation structure and realises particularly experimental actions bearing a certain risk of failure.  The most appropriate format to be adopted by a project evaluation should either be that of an ongoing evaluation or that of a “classical” mid-term evaluation.  The focus of an evaluation must serve the project & its (financial) scale must be reasonable. Individual project evaluations can elaborate “mini-case studies” which analyse all or certain operational elements of a project(e.g. an entire work package, a work process)or focus on issues of a particular common interest(e.g. depth & quality of project-level co-operation; emerging direct effects & unexpected direct/indirect effects etc.).

11. Thank you for your attention!