X ING Huaibin SHI Xiaoyong China National Center for S&T Evaluation (NCSTE)

1. Recent Experiences and Challenges of Research Program Evaluation in China: An Introduction to the International Evaluation of NSFC Funding and Management Performance XING Huaibin SHI XiaoyongChina National Center for S&T Evaluation (NCSTE) 25 October 2012, Minneapolis

2. THIS PRESENTATION • Provides a successful case of R&D program evaluation in China which has been done from 2010 to 2011 • CASE: International Evaluation of NSFC’s Funding and Management Performance • To show some recent experiences and challenges

3. ABOUT NCSTE • Founded in 1997, more than 50 staff now. • www.ncste.org

4. OUTLINE • Background • Methodologies • Implementation • Future and Challenges

5. Background Why to conduct this evaluation? ■ Performance of the government, including of the public funding on R&D have recieved more corcern in recent years in China ■ 2011 marks the 25th anniversary of NSFC ■ The Ministry of Finance and NSFC jointly commissioned this evaluation in Feb, 2010

6. Background Purpose of This Evaluation ■ The Evaluation will, from historical angles, comprehensively examine NSFC’s funding and management performance, its strategic positioning over the past 25 years, and from an international perspective, assess its strengths, weaknesses as well as the challenges it faces.

7. OUTLINE • Background • Methodologies • Implementation • Future and Challenges

8. Methodologies Firstly,domestic preparation, international evaluation. Secondly, Issue and Key Question-Oriented Thirdly, Evidence-Based

9. Methodologies Firstly，“domestic preparation, international evaluation”. □Domestic Preparation NCSTE, an independent and professional evaluation organization,will design the overall evaluation plan and prepare evidence needed for the Evaluation, with the support of NSFC. □International Evaluation International Evaluation Committee (IEC) will evaluate the overall performance of NSFC’s funding and management based on the evidence prepared by NCSTE and its own investigations and international comparisons

10. Methodologies Why need domestic preparation ？ ■The Evaluation covers a long time span and a thorough understanding of Chinese contextare needed; ■There are no systematic evaluation reports on NSFC; ■Source materials are mostly in Chinese. Useful information needs to be selected from large amount of materials. ■Most workshops and interviews should be implemented by NCSTE.

11. Methodologies domestic preparation conducted independently Independence of domestic preparation is reflected in ■Institutional arrangement in NSFC ■Ensure NCSTE team had full independence to conduct their work without interference ■NSFC provided coordination and support request by NCSTE team ■ NCSTE team have the independence to: ■Access to all relevant information ■Select the target group for interview or focus group meeting ■Conduct Questionnaire surveys ■Present evidences and evidence based findings ■Communicate with the IEC directly

12. Methodologies Secondly, Issue and Key Question-Oriented 12

13. Methodologies Thirdly, Evidence-Based Ensure that the IEC has an adequate evidence base available for future evaluation Focus on answers to the evaluation questions defined in the TOR Both qualitative and quantitative evidences; multiple approach for evidence collection

14. Methodologies Evidence-Based 14

15. Methodologies Evidence Collection ■Casesprepared by the NSFC ■Other evidenceprepared by the NCSTE ■Review of relevant policy documents ■Analysis of NSFC documents and statistics ■Bibliometric study on the SCI paper with NSFC ■Interviews and focus groups with government officials, beneficiary institutions, Principal Investigators, peer reviewers and NSFC personnel ■Questionnaire surveys of supporting institutions, Principal Investigators and peer reviewers of NSFC projects ■Case analysis base on the Case Collection of the NSFC

16. OUTLINE • Background • Methodologies • Implementation • Future and Challenges

17. organizational structure of the Evaluation Leading Group Working Group IEC Evaluation Office NCSTE Implementation

18. Terms of Reference cases report 10 issues study reprots SCI papers report Questionnaire Survey report Interviews and Focus Group Meeting report Synthesis Evidence Report International Evaluation Report Implementation Files and Reports

19. Implementation Interviews-4 types

20. Implementation Focus Group Meeting-7 types

21. Implementation Questionnaire Survey- 6 types

22. Implementation Contents and structure of the Synthesis Evidence Report by NCSTE ■Full coverage of the requirements defined in the ToR ■The logical framework: 4 dimensions, 10 key issues,30 questions and 96 evidences ■The structure: Part A (Introduction), Part B (Background), Part C (Key issues and Evidences) and Annex

23. The Evaluation Report and Synthesis Evidence Report can be download at • http://www.nsfc.gov.cn/nsfc/cen/pgbg/index.html

24. OUTLINE • Background • Methodologies • Implementation • Future and Challenges

25. Future and Challenges • Evaluation increasingly received attention from central government but is still a lower priority. • Evaluation of R&D programs and projects has not been institutionalized in related ministries and agencies. • Evaluation results are seldom used in the research priority setting. • Lack of indicators to reflect the performance of different research program. • ……

26. Many thanks for your attention!