1 / 42

How a long-term perspective in evaluation contributes to policy learning - some experiences

How a long-term perspective in evaluation contributes to policy learning - some experiences. Torbjörn Winqvist VINNOVA Evaluations. Agenda. Demands for impact analyses VINNOVA’s approach to impact analyses 3 examples some reflexions. MAIT Seminar 2001 – Dutch Ministry of Economic Affairs

mgeil
Télécharger la présentation

How a long-term perspective in evaluation contributes to policy learning - some experiences

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How a long-term perspective in evaluation contributes to policy learning - some experiences Torbjörn Winqvist VINNOVA Evaluations

  2. Agenda • Demands for impact analyses • VINNOVA’s approach to impact analyses • 3 examples • some reflexions

  3. MAIT Seminar 2001 – Dutch Ministry of Economic Affairs • Issue: Which is the best way to stimulate innovation in the Netherlands? • Empirical studies of governance in 12 countries • Seminar in June 2001 • members of Dutch parliament • evaluation expertise (Europe, US, Australia) • Conclusion: • Evaluators - State-of-the-art evaluation methodology has limitations • MP - We need feedback – if we do not receive it wemake decisions anyway

  4. Demand for impact analyses • Swedish Ministry of Industry, Employment and Communications demands two impact analyses per year: • ’ analyse impact of it’s activities within two areas. Assess impact in terms of sustainable growth and renewal resulting from VINNOVA’s RTD financing’ • Today I intend to describe VINNOVAs approach to meet this challenge

  5. About VINNOVA - The Swedish national innovation agency Mission: to promote sustainable growth by financing RTD and developing effective innovation systems in the fields of technology, transport and working life Annual budget: Around 160 million euro (2006) No of employees: Around 180 persons Instruments: R&D prgms, CCs, institutes, SME schemes, international co-operation, VINNVÄXT RIS initiatives and others N:o of running programmes: Around 50

  6. VINNOVA’s objectives for evaluations Operational – QA, learn from experience, improve performance (mostly mid term evaluations) Strategic – inform top management on strategic issues (R&D actors, instruments, research areas etc) Existential – inform on impacts of VINNOVA’s actions to the world and to ourselves

  7. VINNOVA’s impact analyses • 4 pilot studies in 2001 • 2 impact analyses concluded • Competence centre programme 1995 – 2003 • Impacts of neck injuries research 1985 - 2003 • 4 analyses under way • Traffic safety research 1973 – 2005 • User oriented ICT research 1984 – 2005 • Learning in working life research 1985-2005 • SME-oriented R&D programmes 1993 – 2005

  8. Example 1 Impact of Swedish Competence Centres Programme 1995 - 2003

  9. University - Industry Centre Programmes- Models and International Influences NSF Engineering Research Centres, ERC, USA 1985 - Cooperative Research Centres, CRC, Australia 1990 - In European countries (examples) Competence Centres, Sweden 1995 – K plus Centres, Austria 1998 – Competence Centres, KKK, Hungary 2000 – Competence Centres, Estonia 2003 – MAP Multi-Actors and Multi-Measures Programme - 2002-2004

  10. Swedish Competence Centres Programme Objectives • To create academic, multidisciplinary Centres of Excellence by actively involving a number of companies in joint research • To promote the introduction and implementation of new technology and to strengthen the technical competence in Swedish industry, mainly through the Centres´ industrial partners

  11. Swedish Competence Centres Programme • 28 Centres at 8 Universities • 220 companies are participating actively • 50 of them are engaged in more than one Centre • 1/3 of them are high-tech SMEs • Governmental partners: • VINNOVA (programme manager; 23 Centres) • Swedish Energy Agency (5 Centres) Long-term commitment (1995-2006) Stage-by-stage funding and evaluations

  12. Evaluation strategy - Competence Centres Programme contract contract contract contract Phase 1 year 1-2 Phase 2 year 3-5 Phase 3 year 6-8 Phase 4 year 9-10 Call (2 steps) International evaluation CC-experts (Good start?) International evaluation CC-experts + peers International evaluation CC-experts + peers Panels Impact analysis 1995 - 2003

  13. Impact analysis Technopolis Ltd: Erik Arnold, John Clark, Sophie Bussillet Evidence Evaluations 1997, 2000, 2003 International comparison (US, Ireland, Canada, Austria, Australia) Questionnaire to participants Interviews with CC managements, boards, researchers, companies, university rectors Case studies of CC Portraits of 5 big company participation (e.g. ABB, Volvo) Success stories

  14. Conclusions • University – industry relationship • Introduction at universities of new, more productive forms for • organisation of research and cooperation with industry • Scientific success • scientific quality and relevance very high (11), high (12) • 17 centres are internationally visible, some 5 considered world leading • 200 (400) new PhDs and >200 licentiates, well trained for work in • industry • Industrial success • highly valued networks • CCs have influenced on multi-national companies location of R&D • innovations producing incomes equal to total prgm cost (2003) • (350 PhDs employed by industry in 2005)

  15. Lessons Industry participation from start – most important success factor Importance of long term financing – it takes years before networks crossing organisational borders function and become productive Relationship academy – industry: Attitudes have changed, barriers have been overcome. Cooperation on equal terms -> mutual confidence International evaluations – excellent tool to develop centres and programme

  16. Process to make results understood Bertil Andersson, CE European Science Foundation – Swedish CC prgm came out favourably in international comparison Evaluator asked to make personal presentation of results to minister on research (prior to bill on research) Results presented in co-operation with members of parliament Competence Centre Day every year +++

  17. Policy learning • Government - understanding of what is a CC, what it does produce • CEs given priority in government’s bill on research in 2005 • 14 Linné centres 2006 – 2016 (3 research councils) • 4 Berzelii centres 2006 – 2016 (Swedish Research Council, VINNOVA) • 25 VINN Excellence centres 2006 – 2016 (VINNOVA) • VINNOVA – understanding of CC instrument strengths and limitations • VINNOVA - Awareness of strengths and limitations of the CC instrument • 8 Institute Excellence Centres 2006 – 2016 (SSF, KKS, VINNOVA)

  18. Example 2Impacts of neck injuries research at Chalmers University of Technology 1985 - 2003

  19. About whiplash injuries • Causes 65% of all disabling road traffic injuries • 2000 p disabled annually • 200 p early retreats annually, incl. younger persons • Injuring mechanisms still not fully understood • Occurs at low speed – e.g. peak hour accidents • Solution –> find better vehicles – not a road environment issue • Worrying increase in number of injuries

  20. Objects of impact analysis Basic research on understanding of whiplash causes New approach within biomechanics (Per Lövsund) VINNOVA (TFB, KFB) 1985-2003 Applied R&D on whiplash prevention solutions Programme on Vehicle Research (PFF) 1994-2003 (Co-operation state – automotive industry)

  21. Impact on companies Increased sales • Impact on society • reduced health care costs • reduced loss of worktime • reduced human pain Upstream approach Impact on research Academic career Importance of VINNOVA/PFF funding

  22. Evaluators Institute of Traffic Economics, Oslo, Norway Prof Arild Hervik, University of Molde, Norway

  23. Combination of three methods • socio–economic analysis • achieved benefits to society vis-à-vis costs for R&D • analysis of expected future company profits • (discounted future cash-flow that market expects future R&D to create) • evaluation of research group at Chalmers • benchmarking with other research groups • Evidence: R&D results, interviews and evaluators own competence

  24. Impact • Socio-economic impact • Volvo & SAAB cars 1997- • 165 MEuro Sweden • (1800 MEuro USA) • Autoliv YSAB System • 165 MEuro • Autoliv crash curtains • 175 MEuro • Volvo SAAB 50% safer • Crash curtain 45% • Impact on safety standards Impact on business Increased sales due to improved safety SAAB, VOLVO Autoliv Crash curtain >1100 MEuro (90 % exports) Centres of excellence Ford and GM Impact on research World class research Centre of excellence GM

  25. SAAB SAHR-system Autolivs YSAB – system for cars already in use Chalmers test equipment and applications - simple products based on advanced understanding Chalmers BioRID test model Autoliv/Volvo - WHIPS active whiplash protection

  26. Network analysis Insurance Institute for Highway Safety Swedish National Road Administration Swedish Defence Research Agency Sahlgrenska hospital Folksam research Autoliv VINNOVA whiplash research Volvo Chalmers PFF EU FP SAAB Auto & Motorsport -92 Euro NCAP

  27. 1985 1990 1995 2000 2005 Dialogue between problem-initiated basic research & technology development TFB/KFB/VINNOVA Prgm on Vehicle Research (PFF) Problem initiated –> Basic research produces understanding –> Development meets problems -> New needs for basic research -> etc Strong cluster - considerable mutual openness CTH – Autoliv – Volvo AB - Volvo PV – SAAB – Folksam – Road Adm - Health care

  28. Conclusions • Basic research initiated by an important societal need • Multi-discipline research • Mutual dialogue university research – industrial development • Cluster within automotive industry marked by openness • Condition for establishing research at Autoliv • VINNOVA’s grants crucial

  29. Policy learning Well received by ministry of industry Useful argument vs ministry of finance and ministry of research Key elements: Format easy to understand Illustrates key policy issues

  30. Comments made by VINNOVA head of transport department • The impact analysis • is used as an example on how society benefits from research - also • outside transport area • has strengthened the trust in VINNOVA and it’s way of operating • has given confidence to VINNOVA and to the automotive industry • – they refer to the analysis • illustrates in a better way than before what VINNOVA strives for • (establishing cooperation, the triple helix approach) • has raised internal interest for impact analyses, also for evaluation and monitoring • has strengthened the position of the researchers at Chalmers, given more confidence • in themselves, a reward for their efforts

  31. Example 3Traffic safety research 1973 – 2005 ongoing

  32. Killed in road accidents in Sweden 1970 - 2004

  33. In 1970 – 2004, traffic volume has trebled number of killed reduced by 85% 1 killed : 10 severely injured : 100 injured

  34. Is it possible to assess impact over a 32 year time perspective ?

  35. Approach Seek opinions of particularly informed individuals Pre-study giving overview of R&D 1949 – 2005 - focus on people (not money, projects etc) Main study using best competence available

  36. Objectives Describe impact on society, industry and research (triple helix) Describe importance of public funding to observed impact Evaluators Institute of Traffic Economics, Oslo, Norway Prof Arild Hervik, University of Molde, Norway

  37. VTI? VV? C T H 6,5 U U 5,2 L T H 5 Impact analysis on three levels 1973 – 2004 49 million euro TFD + TFB + KFB + VINNOVA ffp V T I 11,7 Others 21,2 28 + 21m euro Case studies Year 2000 price level

  38. networks networks knowledge New Diffusion of research in society Publications, conferences, PhD:s patents etc Quantitative measurements Effect chain model Public R&D financing Uppsala - Lund – VTI - Chalmers Academic impact Impact at users General impact Mobility Raised competence Etc Qualitative measurements Public users Private users R&D-based actions in companies Measurements of benefits Importance of public funding Socio-economic analyses R&D-based actions Impact on traffic safety & accidents Socio-economic analyses

  39. Comment Main study is under way. We are optimistic about the outcomes of the analysis.

  40. Reflexion (1) Are impact analyses over longer time periods possible to do? Unknown territory. VINNOVA successful. So far. Have we chosen too simple success stories? E.g. ongoing analysis on User oriented ICT research 1984 – 2005 meets difficulties measuring socio-economic impact (work life research difficult to measure) Complements ‘ex post’ evaluations - often concluded before impact is visible Key factor to communicate results in a way that policy makers can understand Conclusion - at least we can search for the opinions of particularly informed individuals – far better than not doing it - ref MAIT seminar

  41. Reflexion (2) Should impact analyses regard particularly important issues? E.g. ICT 1980 - 2005, biotechnology 1980 - 2005? Or should we focus on mechanisms, instruments, how instruments complement each other? How large studies are possible to manage? Our impact analyses have been time consuming and fairly costly They depend on availability of expertise – that policy makers will respect Are they done elsewhere?

  42. Contact info Torbjörn Winqvist VINNOVA Email: torbjorn.winqvist@vinnova.se Phone: +468 473 3108

More Related