INTOSAI Working Group on Key National Indicators

1. SAI’s role in development and use of key indicators for R&D evaluation: a quantitative example and some concluding remarks INTOSAI Working Group on Key National Indicators Ville Vehkasalo & Timo Oksanen, 23.4.2013, Krakow

2. Presentation outline • Our stance on indicator development • Example of how to use key indicators in quantitative R&D evaluation • Qualitative evaluation possibilities • Concluding remarks; incorporation into the White Paper on KNI

3. About SAI’s role in indicator development • Depending on the national mandates, the SAI’s role can be active or passive – or something in between • However, an active role in indicator development can endanger SAI’s independency and objectiveness • The NAO of Finland has not participated in Finland’s KNI development • Therefore, we have kept an outsider’s view to Finnish KNI-system

4. Example: how can we use key indicators in quantitative R&D evaluation? • EU’s Regional Development Fund (ERDF) aims to achieve the following objectives in 2007–2013: 1) to enhance regional R&D and innovation capacities 2) to stimulate innovation and entrepreneurship in all sectors of the regional and local economy 3) to promote entrepreneurship, in particular by facilitating the economic exploitation of new ideas and fostering the creation of new firms. Source: Regulation (EC) No 1080/2006

5. Example • Cost of the ERDF program in Finland, 2007–2013: 1,7 billion euros (EU funding) • The effects of ERDF program are monitored using these indicators: 1) number of new firms 2) number of jobs 3) unemployment rate 4) employment rate

6. Example 5) regional GDP increase relative to the whole economy 6) share of exports in firms’ turnover 7) share of R&D activities in GDP 8) average educational level. Source: ERDF Program of Southern Finland 2007–2013

7. Example • The number of new firms is included in Finland in Figures, which contains key statistical data about Finland on 25 different statistical topics, produced by Statistics Finland • This statistic is not included in Findicator, the official key indicator compilation

8. Example • How can we measure the effects of the 2007–2013 ERDF program in Finland? • Counterfactual: what would have happened without the program? • We need a control group that was not subjected to the program • But in 2007–2013, the whole country is included in the ERDF program

9. Example • However, in the earlier ERDF program, 2000–2006, small parts of Southern Finland were not includedin the program • Therefore, we can compare the development in these new municipalities to those in Southern Finland that had been included earlier (old municipalities), in order to control for economy-wide fluctuations that may also affect start-ups • Population changes can be accounted for by using per capita figures

10. Example • Straightforward comparison is out of the question, as old and new municipalities have systematic differences: new firms per 1000 capita, population-weighted means year 2005 year 2011 old municipalities 5,04 5,25 new municipalities 7,05 7,37 • Even before joining the program, new areas had higher rates of firm creation

11. Example • In order to control for unobservable characteristics, we have to use panel data: the same municipalities before and after the policy change • Specifically, we use the number of new firms from 2005 (before) and 2011 (after) in each of these municipalities • Small sample: only 31 new municipalities vs. 34 old ones (N = 65)

12. Example • We use the difference from 2005 to 2011, Dy =y2011 – y2005, as the independent variable • Differencing wipes out time-invariant characteristics, such as proximity to a larger city • Regression Dy = a + b new_munic • Coefficient estimates are: coef. robust s.e. p-value new_munic-0,095 0,352 0,788 constant 0,150 0,200 0,455

13. Example • new_munic estimate has wrong sign but it is statistically insignificant • Previous estimates are unweighted, i.e. small and large municipalities get the same weight, or importance, in the results • Alternatively we can use weights that measure the size of the municipality, for instance population levels

14. Example • If we use 2005 population levels as weights, we get these estimates: coef. robust s.e. p-value new_munic0,1100,1710,525 constant0,2050,1230,101 • Again, can not reject null hypothesis

15. Example • Average change of +0,31 in the intervention group differs from zero (p = 0,014) but it would be misleading to attribute this to the program • We had an average change of +0,2 in the municipalities that were included earlier, i.e. even without this “new” program • The ERDF program did not cause the observed increase of 0,31 in the number of new firms

16. Example • This example is a bit unrealistic (sample too small, etc.) but it illustrates the basic quantitative evaluation framework: 1) Gather relevant data on intervention and control groups, before and after the intervention 2) Use simple difference-in-differences regression or standard panel data methods 3) Present your results with careful interpretation

17. Qualitative methods • Quantitative methods are useful in assessing program effectiveness • In addition, there are various qualitative approaches to R&D evaluation, such as interviews and participant observation • Possible explanations to why or how something happened/did not happen as planned • General conclusions not possible

18. R&D subproject conclusions (1): Evaluating specific programs and interventions • Evaluation of R&D programs is difficult, but not impossible • Finding relevant data can be tricky • Not possible to evaluate all programs; must have control groups • Without proper analysis, indicators are of limited use in program evaluation

19. R&D subproject conclusions (2): Evaluating the whole R&D system as a part of modern society • Problems are threefold: normative, causative and conceptual • Lack of clear, strategic whole-of-society vision communicated by the government (normative) • Lack of understanding and knowledge about the general impacts of R&D system on modern economies (causative) • What would and could be the role of SAIs and Key National Indicators of R&D in all of this? (conceptual)

20. R&D subproject: Incorporation into the White Paper on KNI • WG Secretariat can freely use our reports in preparing/editing the White Paper on KNI • For instance, our reports could be useful in augmenting the section Principles and Guidelines, subsection Guidelines for knowledge-based economies, where the evaluation of R&D programs is already mentioned

21. R&D subproject: List of reports • Utilising R&D knowledge at R&D policymaking in Finland: problems and promises, Helsinki 2011 (.ppt) • SAI’s role in development and use of key indicators for R&D evaluation, Riga 2012 (.ppt) • SAI’s role in development and use of key indicators for research and development (R&D) evaluation, 2012 (.doc) • SAI’s role in development and use of key indicators for R&D evaluation: a quantitative example and some concluding remarks, Krakow 2013 (.ppt)

22. Thank you! • ville.vehkasalo@vtv.fi • timo.oksanen@vtv.fi • http://www.vtv.fi/en