Stakeholder Involvement and the Choice of Science & Technology Policy Outcome Evaluation Methods

1. Stakeholder Involvement and the Choice of Science & Technology Policy Outcome Evaluation Methods Presenter：Shan-Shan Li STPI, NARL Presentation on 2009 AEA conference 2009.11.14

2. Outline • Introduction • Methodology • Analysis • Research Fundings • Conclusions

3. Introduction

4. The importance of stakeholder • Stakeholder involvement plays an indispensable part in science & technology management. • Stakeholder involvement can… • help to clarify the root causes of problems and to develop effective strategies for addressing them during different phrases of the S&T management process • Evaluation methods are the means for solving such problems or meeting stakeholders’ requirements.

5. Comparison of 10 globally used policy outcome evaluation methods Sources: Summarized and organized from Fahrenkrog, Gustavo, Polt, Wolfgang, Rojo, Jaime, Tubke, Alexander, & Zinocker, Klaus. (eds.) 2002

6. Research questions

7. The external environment: This is formed by species elements and is related to the S&T management including technology, law, economy, politics, physics, labor supply and demand, culture and environments, etc. External environment Internal Environment Profit Organization Competitors The internal environment: which is the environment that is related to S&T management and assignment, used to describe individuals or a group of stakeholders that participate directly in the internal S&T environment, such as customers, contractors, etc. End user Industrial strategic Partners Researchers Mass media Administrative personnel Reviewers Customers/community Research team Consultants Contractors Government/decision makers S&T Funding Organization Stakeholders in S&T management

8. Methodology

9. Allocation of funding Science community’s research output, Outcomes and impact Research fields’ collaboration societal impact (impact of research utilization) the end-users utilizing the final products or services of S&T research A cooperative institutions A S&T executive institutions A the end-users utilizing the final products or services of S&T research B funders A cooperative institutions B the end-users utilizing the final products or services of S&T research C S&T executive institutions B cooperative institutions C the end-users utilizing the final products or services of S&T research D S&T executive institutions C funders B cooperative institutions D the end-users utilizing the final products or services of S&T research E Methodology we will use the 4 phases explained in the Academy of Finland’s report to further derive the issues that the stakeholders are concerned with in order to draw a strategic plan for the future Sources: Academy of Finland (2006)

10. The properties of the research funding impact phases Sources: Academy of Finland (2006)

11. The 4 groups of stakeholders within S&T management • Government • Investigator • Contractor • Competitor • Decision-maker Sponsors S&T R&D management environment S&T R&D implementation system+ Cooperative network system • Research team • Administrators • Consultants • Profit Organization (e.g. insurance) • Academic system • Industrial strategic partner S&T R&D result users • End users • Customers • Academic staff • Community

12. Stakeholders’ supply-side and demand-side

13. Analysis

14. Analysis • Through the analysis of the two dimensional matrix of ： • stakeholders’ interests • evaluation methods • Comparisons (demand-side and supply-side) are shown in p.16, p.17.

15. Direction of discussion : example

16. Comparisons between stakeholders, and evaluation methods (Demand-side)

17. Comparisons between stakeholders, and evaluation methods (Supply-side)

18. Research findings

19. Macroeconomic modeling and simulation It can be applied together to measure policy’s direct outcome and between research fields, inter-industries spillover effects’ direct outcome. supply side demand side The community concern about impact of R&D input on the economy (e.g. economic output and employment creation etc.) The funders’ concern about the externalities problem created by research resources (e.g. knowledge spillover effect or industrial innovation spillover effect etc.)

20. Micro econometric model This method can directly measure the outcome of public policies towards company or individual project. supply side demand side In order to answer questions about rationality of fund allocation and external outcome etc. and also help S&T research and development implementation system into comparing similar projects implementation results, for S&T research and development results users, the model can evaluate researches for instance R&D activities’ achievements (e.g. relationship between R&D budget and company’s ability to earn profit) or spillover effect of individual project’s R&D activity towards industries etc As for the community, evaluation of the effect caused by government’s science and technology policy measures can be done using the method (e.g. implementation and authentication results of scientific projects)

21. Cost benefit analysis It calculates the socio-economic impacts when the implementation plan is carried out, and through the cost benefit analysis result, suitable project, case or policy can be selected. It helps the manager to think about the cost and benefit in a more structural and systematic way. supply side demand side It can evaluate academic staff’s S&T research and development energy development, research output quality and also S&T research and development result users and the community in the level of research output utilization It is suitable in measuring the appropriateness of the resources and funds allocation and the efficiency and the outcome of the project implementation

22. Expert panels/ Peer review They are able to not only carry out ex-post measurement, but also ex-anti evaluation. supply side demand side experts can use the rationality of the expected budget allocation to further analyze the ex-anti evaluation section. It can play an important part in terms of S&T research and development implementation system’s investigation fairness & transparency, communication process and the utilization of the energy development. with research result development and utilization, they can also provide credible and effective suggestions.

23. Control group approaches This method is mainly under the control of other variables, explore the effect single manipulated variable has on different samples; the main manipulated variables are sponsorship funds and applicants. supply side demand side It is able to help provide analytical information on the demand-side’s fund allocation issues and also additionality effect, such that academic staff’s energy development can be further analyzed. It is appropriate to apply control group approaches on measuring and evaluating private enterprises economic activities for example: the impact of governmental department’s S&T research and development grants on private enterprises R&D, the impact of government fiscal measures (lowering taxes), and the effectiveness of loaning between emerging companies and banks etc.

24. Innovation surveys It mainly focus on the evaluation of the corporations’ innovation activities. supply side demand side Stakeholders will be very much concern about economic and industrial growth; level of commercialization and product sales due to innovation etc Funders can monitor whether policy tools can help to achieve product innovation target or not through funds grant formula or rationality of funds allocation

25. Network analysis It is mainly a relationship structure between each node and also the behavioral change result at each node through the end of the decision-making process. supply side demand side It can fully perform its function when looking at the fund allocation communication process and network system cooperation direction on the demand-side. It explores the interaction and state between each node; therefore the academic staff’s S&T research and development energy development and users’ energy enhancement on the demand-side can also be analyzed and explored Cooperation network system’s stakeholders have always been the target of the network analyze explore. As a result, necessary information can be obtained through network analysis; also it can explore the condition of knowledge distribution

26. Foresight/ technology assessment The transfer of foresight and technology onto the evaluation is mainly a comprehensive outline of the whole fund allocation direction and S&T research and development project content. supply side demand side It would be more interested in knowing the safety of the research and development result and whether or not future demand is met etc The stakeholders on the supply-side will care most about the S&T research and development planned implementation procedures and rationality of the allocation of the public funds

27. Field/ Case study The survey method is qualitative and is suitable for those innovative projects that haven’t been analyzed in depth yet and is also suitable on the policy measures that can have complicated socio-economic impacts, though it is not very appropriate to use it to test the causal relationship. supply side demand side In order to respond the S&T impact. It is the application of these issues, Investigation of system fairness and transparency, Cooperation direction, S&T research and development energy development, Research utilization/ safety/ function/ quality, Energy upgrade, Research development utilization/ safety/ function/ quality Due to explore the details of S&T implementation. It is suitable for some issues, Investigation of system fairness and transparency, Cooperation direction, S&T research and development energy development, Research utilization/ safety/ function/ quality, Energy upgrade, Research development utilization/ safety/ function/ quality

28. Benchmarking It can apply on product, service, procedure, method, framework, and the whole organization etc. In the process of benchmarking, target selection is very important. supply side demand side The choices of indicators can be applied in some issues, Investigation of system fairness and transparency, S&T research and development energy development The choices of indicators can be applied in some issues, Investigation of system fairness and transparency, S&T research and development energy development

29. Conclusions • Stakeholders are a determinate element in the choice of outcome evaluation methods. The level of attention paid towards each issue will depend on each stakeholder ultimate goals causing a constant need of coordination in conflicted S&T policy decision makings and evaluation systems. A combination of multiple S&T policy outcome evaluation methods is necessary. • Government organizations can carefully formulate a suitable policy through interacting with the stakeholder ; resolving concerning issues and planning future development of the policies together, in order to achieve a realm of ‘mutual benefit’. • The identification of stakeholders indeed have influence on the whole process of evaluation. • When we make the choices of evaluation methods, we need consider the indispensable element, the stakeholders.

30. Li, Shan Shanssli@mail.stpi.org.twhttp://www.stpi.org.tw/STPI/index.htm

31. Thanks For Your Attention!