Chris Gerrard, Team Leader, PIM Evaluation

1. The IEA’s Approach to Assessing the Scientific Quality of CGIAR Research Programs: Illustrated by the PIM Evaluation Chris Gerrard, Team Leader, PIM Evaluation IAAE Pre-conference workshop on “Measuring the value of policy oriented research” Milan, August 8, 2015

2. The CGIAR’s Independent Evaluation Arrangement (IEA) • Established in 2012, located in FAO, Rome — along side the Independent Science and Partnership Council Secretariat (ISPC) • Responsible for conducting System-level independent evaluations • Has now (2015) completed evaluations of five CGIAR Research Programs (CRPs), with five more ongoing • Available at “http://iea.cgiar.org/evaluations” • What used to be called CGIAR core funding (mobilized centrally) is now being channeled through 15 multi-Center, multidisciplinary CRPs, one of which is PIM (Policies, Institutions and Markets)

3. What is Policies, Institutions and Markets (PIM)? • Based at IFPRI, involving 12 of the 16 CGIAR Centers • Supports evidence-based research to help improve public policies and expenditures for pro-poor, sustainable agricultural growth in developing countries • The CRP with the greatest focus on social science and policy research to achieve the CGIAR’s overall objectives of reducing rural poverty, increasing food security, improving nutrition and health, and more sustainable management of natural resources • Spent $261 million in 2012–2014, of which $68 million (26%) came from centrally mobilized CGIAR resources • Was organized into 7 flagships and an eighth cross-cutting flagship on gender, partnerships, and capacity strengthening

5. The Majority of PIM Research Is Based at IFPRI

6. The Independent Evaluation of PIM Addressed the Following Issues . . . • Program Focus: Is it supporting research with clear and coherent objectives that are responding to global, regional and country development challenges? • Inter-Center Collaboration: Is it creating opportunities for researchers to engage in relevant and effective collaborations across Centers? • Gender: Is it mainstreaming gender perspectives and analysis in PIM research? • Quality of Science • Partnerships and Impacts: Is it fostering strong and innovative partnerships for positive development impacts? • Organizational Performance: Does it have streamlined and efficient governance and management, with clear accountability?

7. . . . And Followed IEA’s Approach and Methodology for Assessing Quality of Science

8. Review of Previous Assessments • Barrett et al., Stripe Review of Social Sciences in the CGIAR, 2009. • The CGIAR was increasingly characterized by short-term planning and reporting cycles that forced researchers to prioritize short-term impacts over sound long-term scientific goals. • The research agenda had become more fragmented and less focused on the areas in which the CGIAR enjoyed a comparative advantage. • Disciplinary diversity was lacking as 60 percent of all social scientists in the System were economists. • Research designs and methods used throughout the research process were deficient in a majority of cases. • ISPC Commentaries on the original PIM proposal (2011) and on the proposal for extension phase (2014) • The gender perspective is very strong throughout the portfolio. • There is little or no research that focuses on the science-policy interface that could help inform and improve PIM’s stated impact pathways. • PIM does not have a clear and convincing long-term data strategy.

9. Quality of Principal Investigators, by Peer-Reviewed Publications and Center, 2013-2014

10. Quality of Principal Investigators, by Career H-Index and Center

11. Reasons Given by Some Researchers for Diminished Productivity • They were managing large and complex projects that involved a great deal of fieldwork and data collection, which had not yet generated results because of these time-consuming activities. • They chose to present the results of their research in a non-peer reviewed format to better target policy makers, practitioners and other decision makers in the country where the research was taking place. • They were doing delivery-type research that sought to provide the research results directly to development actors without delays. It was more difficult to get delivery-type research published in leading journals. • PIM management had not explicitly indicated expected levels of research outputs. Researchers had to discern this implicitly when their funding was discontinued or new proposals were not approved for funding. PIM management appeared to be using IFPRI-based standards which were historically higher than in their own Centers, which did not have the same quality of research support.

12. CGIAR Staff Perceptions of Quality Assurance Mechanisms, by Center

13. Choice of Research Topics, Design and Methods: Portfolio Analysis of 74 Projects

14. Choice of Research Topics, Design and Methods: Survey of CGIAR Partners

15. Qualitative Review of 25 Randomly Selected Publications

16. In-Depth Review of Three Modeling Systems • Few organizations have a similar combination, range, and quality of modeling systems as IFPRI, and therefore PIM • Scientists leading these efforts are high quality. • Foresight modeling based on the IMPACT model • Currently being restructured and enhanced based on an earlier (2010) review • MIRAGE Trade Analysis Model and MIRAGE Biofuels Model • For analyzing the Doha round of WTO negotiations and the EU biofuels policy, among other things • Country-Level CGE Modeling • Widely recognized IFPRI expertise, leading to many requests from countries for analytical support, particularly in Africa

17. Number and Type of Publications, 2012-2014 (both peer- and non-peer-reviewed)

18. Placement in Peer-Reviewed Journals

19. Citation Frequency of PIM Publications, Published in 2013 • 58% of PIM papers were cited less than the average frequency for each journal

20. Additional Alternative Metrics • The number of views and their IP location codes • The number of downloads of the paper and IP location • References to PIM research in mass media (to researcher or paper), classified according to type of outlet • Impact indicators on Researchgate (academic social media platform) • References to PIM research results on Facebook, Twitter, and other social media • References to PIM-supported research at high-level international policy meetings on food security, natural resources, or international development in general.

21. Overall Conclusions • PIM is young program (started in 2012). Therefore, the positive findings in relation to input quality are more convincing than those relating to output quality and impact, which often relate to legacy research that started before PIM. • The program is doing well on the relevance of scientific topics and quality assurance mechanisms. It is doing less well in relation to minimum standards of scientific productivity and impact. • PIM has a good number of very productive researchers whose publications are attracting large numbers of citations. • IFPRI-based researchers have stronger support systems. The increasing degree of inter-Center collaboration has the potential to raise the scientific quality of social science and policy research at the non-IFPRI Centers.