Evaluation of Knowledge Innovation Program (KIP) at Chinese Academy of Sciences

1. Evaluation 2011: Values and Valuing in Evaluation Anaheim, California Logic Model and Methodology for Knowledge Innovation Program(KIP) Evaluation in Chinese Academy of Sciences Tao Dai Nov. 4, 2011 Institute of Policy and Management, CAS CAS center for management innovation and evaluation research

2. Outlines • Background • Logic model for KIP evaluation • Methods for KIP evaluation • Main evaluation results • Conclusion

3. Background • What is KIP? • launched and implemented by CAS • three phases: 1998~2000,2001~2005,2006~2010 • aims: improve the scientific performance of CAS ;make CAS pre-eminent Science and technology (S&T) center • Why do KIP evaluation ? • Chinese government’s requirement • CAS’ requirement

4. Background • Challenges of KIP evaluation • Systematic evaluation on all the work CAS carried out • Cover 13 years • CAS-centered self-evaluation How to ensure the objectiveness and fairness ?

5. Logic model of KIP evaluation

6. Objectives of KIP “By 2010, CAS will have built itself into a knowledge creation center with powerful and continuous innovation capability, aiming at national strategic objectives and international scientific frontier. It will have become a scientific research base with its overall level reaching advanced international standards, and an important base for bringing up high-caliber talents, and a base for promoting the development of high-tech industries in China. It will have developed into a knowledge base, and become the state’s think tank and talent bank with international influences.”

7. Evaluation methods

8. The bottom layer: basic materials and data collected • progress report to the central government during each stage • During the implementation of KIP, hundreds of experts, managerial professionals and users participated in the evaluation and presented precious suggestions and comments • monitoring mechanism

9. The second layer: thematic evaluation 1) Self-evaluation and communicating review of the research institutes • 100 institutes conducted self-evaluation firstly • related materials of self-evaluation were circulated on internet • reviewed by 201 experts from CAS headquarters, bureaus, and institutes

10. The second layer: thematic evaluation 2) Consultant review on S&T innovation base • each innovation base conducted self-evaluation • 132 experts were invited to discuss and review based on the self-evaluation, through meeting review, corresponding review • consultant review committee made the comprehensive review

11. The second layer: thematic evaluation 3) Self-evaluation and summary of key aspects of management work • strategic planning and disciplinary deployment • construction of academic divisions • contingent construction • system reform and construction • resource allocation and asset management • Local and international cooperation • infrastructure construction • …

12. The second layer: thematic evaluation 4) Seminar on deepening the reform of CAS • 3-day seminar • 150 directors from both academy and institute level • focused on the key issues in CAS developments

13. The top layer: Overall evaluation of KIP • carried out simultaneously and interactively with thematic evaluation (the second layer) • made full use of the existing evaluating results and data (the bottom layer) • the report titled ‘The Evaluation Report on KIP (1998-2010)’ was formed and submitted to the state council of China

14. Evaluation Results • Innovation capability improved significantly • Number of high quality papers（Top 1%citation）in 2009 is 11.5-fold increase from 1998 • Undertaking 33%“973 program” and 34% “NSFC Key Project” in 1998~2009 • Operating 80% large-scale scientific facilities of China • …

15. Evaluation Results • Cultivating and Fostering high-level S&T talents • Bring in 1292 outstanding talents from overseas • About 900 scientists held important positions in international organization or international academic journals • Number of Ph.D. graduate from CAS is about 8% of the total in China, contributing to 18.5% of the “outstanding doctoral thesis” • …

16. Evaluation Results • Effective in S&T achievements transfer • Number of transfer programs of S&T achievements in 2009 is 5-fold increase from 1998 • The total Business income (1998~2009) of CAS invested enterprises is about 1,086 billion yuan • The total profit and tax(1998~2009) of CAS invested enterprises is about 52 billion yuan • …

17. Evaluation Results • Very active indomestic and international cooperation • Built 335 technology centers with companies • Built 10 research institutes with local government • Built 2 national labs and 6 national key labs with university • Number of international academic conferences organized by CAS increased from 57 in 1998 to 354 in 2008 • Number of International cooperation and exchange visitors increased from 10 000 in 1998 to 28 000 in 2009 • …

18. Conclusions • feasible to evaluate the performance of CAS in KIP according to the evaluation practice • based on objective data, facts, and experts judgment • qualitative evaluation was combined with quantitative evaluation; self-evaluation was combined with experts review

19. Thanks for your attention!