1 / 25

American Evaluation Association Conference October 28, 2005 Kazuaki Komoto, kohmotokza@nedo.go.jp

American Evaluation Association Conference October 28, 2005 Kazuaki Komoto, kohmotokza@nedo.go.jp Shuji Yumitori, Hiroshi Sano, Takahisa Yano. Follow-up Monitoring and Evaluation Systems for National R&D at NEDO -Part 1- ~Results from follow-up monitoring~.

hedya
Télécharger la présentation

American Evaluation Association Conference October 28, 2005 Kazuaki Komoto, kohmotokza@nedo.go.jp

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. American Evaluation Association Conference October 28, 2005 Kazuaki Komoto, kohmotokza@nedo.go.jp Shuji Yumitori, Hiroshi Sano, Takahisa Yano Follow-up Monitoring and Evaluation Systems for National R&D at NEDO -Part 1-~Results from follow-up monitoring~

  2. Follow-up Monitoring & Evaluation 　　　・ Introduction 　　　・ Outline of the Method Used 　　　・ Activities in FY2004 1) Results of Simplified Monitoring 2) Results of Detailed Monitoring 　　　 ・ Activities in FY2005 1) Results of Simplified Monitoring 　　　 ・ Conclusion of Monitoring 　　　 ・ Evaluation CONTENTS

  3. Introduction This presentation explains the following: 　☞ An outline of the method of follow-up monitoring & evaluation. 　☞ The results of follow-up monitoring and the effects of NEDO’s projects. 　☞ The evaluation of the results of such monitoring.

  4. The objective of this new initiative are: To assess the impact of NEDO’s R&D projects on the technological progress in the relevant fields, by monitoring post-project activities of project participants toward practical applications of R&D achievements, To provide feedback to improve R&D management , by reviewing previous post-project evaluations in view of post-project activities, and To contribute to the planning future of R&D strategies, by understanding technology trends in post-project research activities. Follow-up Monitoring & EvaluationOutline of the Method

  5. Step 2 Detailed monitoring Detailed monitoring based on simplified monitoring Questionnaires or interviews(sales amount, market share, reason for discontinuance, other spin-off effects, etc.) Step 3 Evaluation Implementation under the evaluation committee 1）To assess the impact of national R&D projects 2) Improved R&D management 3) Planning future R&D strategies Follow-up Monitoring & EvaluationProcess Step 1 Simplified monitoring Grasp the current situation of an organization’s activities 1) Basically annually for five years 2) Through data collection 3) In addition, if there are changes in contact person in the organization Key questions: ・ Has this R&D accomplished the final target ?, ・ Has there been delay of 2 years against the pre-set target?, or ・ Are there any other necessity?

  6. Follow-up Monitoring & EvaluationSimplified Monitoring ☞ Target projects All 27 projects completed in FY2001 All 29 projects completed in FY2002 ☞ Target organizations Private companies, research institutes, public entities, universities ☞ Questionnaires For companies 　　 　・Progress of the following activities: ☞ Research ☞ Development ☞ Practical application ☞ Commercialization 　　 　 ・ Paper, awards received, names of products, Far-reaching effects 　　 　 ・ Competition situation (precedence/equal/delay) For universities 　　 　 ・ Research activities Total:56 projects Total:642 organizations -Areas of monitoring- ○ Electronics and information ○ Nanotechnology and materials ○ Biotechnology and medical technology ○ Machinery systems ○ Environment ○ Fuel cell and hydrogen technology ○ New energy ○ Energy conservation

  7. Follow-up Monitoring & Evaluation Results of Simplified Monitoring in FY2004 Response Rate by Respondents ◆ 541 responses have been received to date ◆ Disaggregated data are as follows; *Organizations being monitored Total (84%) Private Companies (95%) Public Entities (96%) Research Institutes (71%) Universities (66%)

  8. Follow-up Monitoring & Evaluation Results of Simplified Monitoring in FY2004 (cont.) Entity (Private Company) Basis Analysis No Post-project Activity* １２０ （３６％） Discontinued １２ (5%) ３３３ Number of private companies Research ７６ (36%) Post-project Activity ２１３ （６４％） Development ７９ (37%) Practical Application & Commercialization ４６ (22%) Post-project Status Total Number of Responses Stage of Post-project Activity *No Post-project activity : No follow-up activities planned after NEDO’s project is completed. 1) 64% of the private companies are currently conducting post-project activities. However, 36% of the private companies have no post-project activities. 2) 22% of the private companies have reached the stage of practical application or commercialization of R&D achievements. New Energy & Industrial Technology Organization

  9. Follow-up Monitoring & Evaluation Results of Detailed Monitoring in FY2004 (cont.) ☞ Motive for participation in NEDO’s project ※ Multiple answers allowed Personnel training Other Challenge a new technical target Practical application & commercialization (n=78) Invitation of other organization Matched a main purpose of the organization Improvement of technology Create network No post-project activity (n=246) Challenge a new technical target Invitation of other organization Matched a main purpose of the organization Improvement of technology Create network Personnel training Other ◆Motivations of companies carrying out practical application and commercialization matched with a main purpose of the company. ◆ Companies with no post-project activity aimed to challenge a new technical target and to improve the level of technology.

  10. Follow-up Monitoring & Evaluation Results of Detailed Monitoring in FY2004 (cont.) ☞ Timing of technical seed occurrence Other (4%) Before NEDO’s project started (30%) During NEDO’s project (29%) Before NEDO’s project started (67%) During NEDO’s project (70%) (n=48) (n=10) Practical Application & Commercialization Discontinued the Post-Project Activity ◆ Companies in the practical application and commercialization stage had developed technology seeds before the project started. ◆ However, most companies which have discontinued the post-project activity had tried to develop the technical seeds in the project phase.

  11. Follow-up Monitoring & Evaluation Results of Detailed Monitoring in FY2004 (cont.) Growth of participating R&D departments Establish a related R&D project No effect Others Effect on R&D policy of companies Effect on R&D activities of companies (n=41) Contribution to establishment of a new enterprise (n=41) (n=38) ◆After achieving practical application or commercialization, 37% of monitored companies established new enterprises such as new R&D departments, spinoffs and/or joint ventures. ◆Growth of participating company R&D departments was recognized in 32% of monitored companies, and related R&D projects have been launched in 29% of monitored companies.

  12. Follow-up Monitoring & Evaluation Results of Detailed Monitoring in FY2004 (cont.) Contribution to job creation Impact on academic field (n=38) (n=36) ◆More than 90% of monitored companies recognized some effect on job creation after achieving practical application or commercialization. However, the effect was limited. ◆More than 70% of monitored companies recognized that the results obtained from NEDO’s R&D project had some effect on the academic field, and 34% answered it was significant. New Energy & Industrial Technology Organization

  13. Follow-up Monitoring & Evaluation Results of Detailed Monitoring in FY2004 (cont.) Improvement of technology /products Support for high-risk R&D Formation of network of personal contacts Improvement of quality of researchers Other/Nothing Special ※ Multiple answers allowed BeforeNEDOproject DuringNEDOproject AfterNEDOproject Expectations regarding NEDO’s R&D Project ◆Expectations of monitored companies regarding NEDO’s project were different before and after the project. ◆Before NEDO’s project, company expectations were focused on technology improvement and/or product performance, as well as engaging in high-risk R&D. ◆However, during the project, companies indicated that NEDO’s project was useful for technology improvement and/or product performance, as well as formation of a network of personal contacts. ◆After completing NEDO’s project, companies saw providing support for high-risk R&D as the most important role of NEDO’s projects. New Energy & Industrial Technology Organization

  14. Preliminary results Follow-up Monitoring & Evaluation Results of Simplified Monitoring in FY2005 Number of Projects and Entity Monitoring in FY2005

  15. Preliminary results Follow-up Monitoring & Evaluation Results of Simplified Monitoring in FY2004 & FY2005 Private company monitoring completed in FY2001 & FY2002 No Post-project Activity No Post-project Activity ◆ Four companies with no activities in 2004 started post-project activities in 2005. ◆ Differences between 2004 and 2005 regarding the stage of post-project activities are explained in the next chart. Discontinued Unanswered Research Number of private companies Post-project activity Research Post-project activity Development Development Practical Application Practical Application Commercialization Commercialization Stage of Post-project activity Stage of Post-project activity Post-project status Post-project status Total number of responses Monitoring in FY2004 Monitoring in FY2005 Progress of Post-project Activities of Private Companies

  16. Preliminary results Follow-up Monitoring & Evaluation Progress of Post-project Activity at Private Companies Monitored in FY2004 & FY2005 ◆Relatively many companies , that is 22% of the total reach the practical application stage or the commercialization stage. ◆ Out of 69 companies in the research stage and 67 companies in the development stage in the FY2004, only 2% went up to the practical application and commercialization stage in FY2005 ◆ Progress of post-project activities toward practical application and commercialization stage does not necessary correlate with the time, meaning there is no linear relationship between time and progress of activities. Progress of Post-project Activities of Private Companies

  17. Preliminary results Follow-up Monitoring & Evaluation Results of Simplified Monitoring in FY2005 (cont.) Private company monitoring Completed in FY2003 300 No Post-project activity There is no significant difference between FY2003 and FY2001/FY2002 81 200 Unanswered 23 企 Number of private companies Discontinued 15 業 Post-project activity Research 255 数 41 174 100 Development 63 Practical Application 17 Commercialization 15 0 回答数 継続事業の有無 継続事業の内訳 Total number of responses Post-project status Stage of post-project activity Progress of Post-project Activities of Private Companies

  18. Preliminary results Follow-up Monitoring & Evaluation Preparation forSimplified Monitoring in FY2005 Private company monitoring Completed in FY2004 100 ◆ More than 70% of companies have plans to carry out some post-project activity. ◆ 43% of these companies plan to continue to attain practical applications or commercialization. No Post-project activity 19 Number of private companies Unanswered 8 企 Research 業 Post-project activity 15 80 数 Development 59 14 Practical Application Commercialization 22 0 Target of post-project activity 回答数 継続事業の有無 継続事業の目標設定 Total number of responses Post-project status Targets after a Project

  19. Preliminary results Follow-up Monitoring & Evaluation Results of Simplified Monitoring in FY2005 Industry-Academia Collaboration ◆After the completion of NEDO projects, half of the university participants still collaborate with private companies, half of which participate in new NEDO projects. ◆ These results indicate that NEDO’s projects may provide a good opportunity for collaboration between industry and academia. Monitoring completed in FY2003, 2004 & 2005 Yes Private companies participating in NEDO projects No 58% Other private companies 42% 52% 48% Collaboration after NEDO’s Project (Responses from university participants)

  20. Follow-up Monitoring & EvaluationConclusion of monitoring (1/2) Follow-up monitoring of post-project activities to find out the effect of NEDO projects 1) For FY2001 & FY2002 projects, 64% of private companies are carrying out post-project activities, and 22% of them have reached practical application & commercialization. There is no significant difference between FY2003 and FY2001/ 2002. 2) Expectations of monitored companies regarding NEDO’s projects ☞ Before : Improvement of technology, challenge high-risk R&D ☞ During : Improvement of technology, Formation of a network of personal contacts ☞ After : Promotion of high-risk R&D

  21. Follow-up Monitoring & EvaluationConclusion of monitoring (2/2) 3) The progress of post-project activities of private companies monitored in FY2004 and FY2005 is : ☞ Relatively many companies , that is 22% of the total reach the practical application stage or the commercialization stage in FY2004. ☞ Out of 69 companies in the research stage and 67 companies in the development stage in the FY2004, only 2% went up to the practical application and commercialization stage in FY2005. ☞ Progress of post-project activities toward practical application and commercialization stage does not necessary correlate with the time, meaning there is no linear relationship between time and progress of activities.

  22. Follow-up Monitoring & EvaluationEvaluation Assess the impact of government-supported R&D projects As a result of monitoring of the post-project activities of project participants, we were able to clarify the effect of NEDO’s projects. Improve R&D management This will be described in Part 2. Plan R&D strategies Continuation of monitoring and data collection.

  23. Thank you very much for your kind attention!

  24. N O T E

More Related