Refereeing and Discussant Guidelines

1. Refereeing and Discussant Guidelines Susan Godlonton AGRODEP AIEN III Workshop Dakar, Senegal 4th June, 2014

2. Referee Report • Components of a referee report: • Cover letter • Summary • Key concerns • Decision • Report • Overall view • Main concerns • Smaller concerns • No “one-size-fits-all approach”

3. Referee Report: Good paper? • What is a good paper? • Clear • Coherent (understandable) • Correct • Additional contribution • Value added of answer to question • Credibility of identification strategy • Validity and robustness of approach

4. Referee Report: Overall view • Don’t repeat the abstract, the editor can read the abstract • How would you summarize the paper • What do you think is the key contribution • Do you think the authors address it correctly

5. Referee Report: Main Concerns • Key issues that the author needs to address • Ensure that it is clear which of your concerns are fundamental • Don’t ask the author to write a different paper • Be constructive! Try to provide solutions where possible accounting for data limitations

6. Referee Report: Main concerns • Related literature • Inappropriate to the actual material in the paper • Inaccurately described • Incomplete (very common) • Logical argument • Incorrect application of economic concepts • Inaccurate mathematical derivations • Loose ties between the economic model and the empirical analysis

7. Referee Report: Main concerns • Econometric tools are being used inappropriately • Incorrectly used • Assumptions do not hold • Assumptions not defended with evidence • Estimated key coefficients of interest • Not properly identified • Not robust • Additional robustness checks needed • Interpretation of empirical results is inappropriate • Overstates the contribution • Overstates the claim (e.g. causation vs. correlation) • Conclusions are incorrectly made or expressed

8. Referee Report: Main concerns • Econometric tools are being used inappropriately • Incorrectly used • Assumptions do not hold • Assumptions not defended with evidence • Estimated key coefficients of interest • Not properly identified • Not robust • Additional robustness checks needed • Interpretation of empirical results is inappropriate • Overstates the contribution • Overstates the claim (e.g. causation vs. correlation) • Conclusions are incorrectly made or expressed

9. Referee Report: Minor concerns • Do not bring these up when discussing a paper • Specific places in text where discussion hard to follow or confusing • Not using standard notation • Spelling and grammatical errors • Missing data sources and poorly constructed Tables or Figures • References to the literature that are missing or incorrect

10. Additional Resources • Refereeing • http://blogs.worldbank.org/impactevaluations/how-much-to-referee-and-how-to-do-it • http://marcfbellemare.com/wordpress/2012/01/contributing-to-public-goods-my-20-rules-for-refereeing/ • http://marginalrevolution.com/marginalrevolution/2006/10/how_to_be_a_goo.html • Discussant • http://chrisblattman.com/2010/02/22/the-discussants-art/ • Other resources will be uploaded to the AIEN workshop

11. For today… • Each been assigned a paper/set of slides from one of your peers • Focus • Clearly define the research question • Define and discuss the identification strategy • Propose alternative credible identification strategy

12. For today… • Focus on the identification strategy • What is the identifying variation? • What assumptions are made by using the chosen identification strategy (implicitly/explicitly)? • Do they make sense? • To what extent are these true? • How else can the author test the assumptions? • What robustness checks or placebo tests could the author run? • What would be a more preferable strategy?

13. Thinking about Identification Homework Assignment

14. Thinking about Identification Exercise • Group into: • Health • Education • Agriculture • Discuss proposed ideas, provide feedback to one another