Peer Review Can Make Metrics Accountable 同行评议可以赋予指标以责任

1. Peer Review Can Make Metrics Accountable同行评议可以赋予指标以责任 Kelli Barr 凯利·巴尔 Department of Philosophy and Religion Studies Center for the Study of Interdisciplinarity University of North Texas 哲学与宗教学系 跨学科研究中心 北德克萨斯大学 Translation by ZHANG Wei

2. The Question at Hand目前的问题 Are metrics better tools for evaluating research than peer review? 在评估研究方面，指标是比同行评议更好的工具吗？ • Are metrics better tools for evaluating research than peer review? • Increasingly, academics and administrators are answering with an unqualified “yes • 专家学者和管理者越来越多地倾向于给出肯定的回答，但这是有问题的。 • H-index, g-index, webometrics H指数、g指数、网络计量学 • US News and World Report, Shanghai Rankings 美国新闻与世界报道，上海（交通大学）大学排名 • National research evaluations 国家研究评估 • Hiring, promotion, and tenure decisions 人才招聘、晋升和终身教职决定 ‘Peer Review Can Make Metrics Accountable’ International Workshop on Peer Review Dalian University of Technology Monday, May 21 – Wednesday, May 23, 2012

3. Peer Review Bibliometrics同行评议 文献计量学 Subjective 主观性 Contingent 差异性 Non-transparent 不透明性 Unaccountable 不可解释性 Costly 高成本性 Objective 客观性 Replicable 可重复性 Transparent 透明性 Accountable 可解释性 Low(er) Cost 低成本性 ‘Peer Review Can Make Metrics Accountable’ International Workshop on Peer Review Dalian University of Technology Monday, May 21 – Wednesday, May 23, 2012

4. Defining the dichotomy using these terms creates a distinction without much of a difference 用这些术语定义的二分法给出了一个其实并没有太多差别的区分 • Bibliometrics are ultimately based on peer review processes 文献计量方法最终要建立在同行评议的程序之上 All metrics instantiate specific values that affect the outcome of the measurements 所有的指标都是某种具体价值标准的体现，都会对评估结果产生影响 ‘Peer Review Can Make Metrics Accountable’ International Workshop on Peer Review Dalian University of Technology Monday, May 21 – Wednesday, May 23, 2012

5. Forbes – concerned with economic, financial, commercial, and business outcomes 福布斯— 涉及经济、金融、商业、贸易的结果 Academic ranking of world universities – concerned with bibliometric (research) outcomes 世界大学的学术排名— 涉及文献计量学的（研究）结果 http://chronicle.com/article/30-Ways-to-Rate-a-College/124160/

6. Journal impact factor, calculated using Web of Knowledge Sciences Nature 34.48 Cell 31.152 Science 29.747 Social Sciences J. of Economic Lit. 6.919 Political Analysis 3. 756 Ann. Rev. of Sociology 3.702 Philosophy Am. J. of Bioethics 4.000 Phil. and Public Affairs 1.957 Environmental Values 1.250 Journal impact factor, calculated using Web of Knowledge 期刊影响因子——基于Web of Knowledge数据库 Sciences Nature 34.48 自然科学《自然》 Cell 31.152 《细胞》 Science 29.747 《科学》 Social Sciences J. of Economic Lit. 6.919 社会科学《经济文献杂志》 Political Analysis 3. 756 《政治分析》 Ann. Rev. of Sociology 3.702 《社会学年评》 Philosophy Am. J. of Bioethics 4.000 哲学《美国生物伦理学杂志》 Phil. and Public Affairs 1.957 《哲学与公共事务》 Environmental Values 1.250 《环境价值》 Do social science and philosophy journals really have that much less “impact”? 社会科学和哲学期刊真的是只有这么点儿影响力吗？ ‘Peer Review Can Make Metrics Accountable’ International Workshop on Peer Review Dalian University of Technology Monday, May 21 – Wednesday, May 23, 2012

7. Bibliometrics assume that all research conforms to the model of scientific research • Cumulative • Disciplinary Bibliometrics highlight certain kinds of research: 文献计量学突出强调某些种类的研究： • Journal publications 期刊发表 • Quantitative 定量的 • Methodologically focused 关注方法论的 • Review papers 评议论文 Bibliometrics assume that all research conforms to the model of scientific research 文献计量学假定所有的研究符合自然科学的研究模式。 • Cumulative 累积的 • Disciplinary 分科的 ‘Peer Review Can Make Metrics Accountable’ International Workshop on Peer Review Dalian University of Technology Monday, May 21 – Wednesday, May 23, 2012

8. Implications 启示 Bias toward disciplinary, status quo research • 对学科的偏见，现状分析 Reinforcing the status quo is not morally neutral • 对现状的加强在道德上不是中立的 Appeals to the supposed objectivity, transparency, etc. of metrics are used as evidence of their normative superiority to peer review 对指标的所谓的客观性、透明性等属性的诉求被用来作为它比同行评议具有规范的优越性的证据。 ‘Peer Review Can Make Metrics Accountable’ International Workshop on Peer Review Dalian University of Technology Monday, May 21 – Wednesday, May 23, 2012

9. Reflexive evaluation of activity toward goals 对面向目标的行动的自反性评价 Phronesis 实践智慧 Rational calculation of goals 对目标的 理性计算 Episteme 知识 Techne 技艺 ‘Peer Review Can Make Metrics Accountable’ International Workshop on Peer Review Dalian University of Technology Monday, May 21 – Wednesday, May 23, 2012

10. Accountability is about rational justification 责任性涉及合理的辩护 • Definitions vary widely with context: • Common denominator: evaluating whether or not current tactics meet overall strategic goals 共同点：评估当前的策略是否能够达到总体的战略目标 Accountability is about rational justification 责任性涉及合理的辩护 • Definitions vary widely with context: • Common denominator: evaluating whether or not current tactics meet overall strategic goals • Accountability is more than just rule following 责任性不仅仅是遵守规则 Accountability is about rational justification 责任性涉及合理的辩护 • Definitions vary widely with context: 定义会因为语境的不同而产生很大的变化： • Legal liability 法律责任 • Return on investment 对投资的回报 • Academic rigor 学术的严格性 ‘Peer Review Can Make Metrics Accountable’ International Workshop on Peer Review Dalian University of Technology Monday, May 21 – Wednesday, May 23, 2012

11. Conclusions: 结论 Neither metrics nor peer review are an unqualified good 指标和同行评议都不是最好的方法 The question of the meaning of accountability is also the question of the meaning of responsibility 责任性的意义问题同时也是一个义务的意义问题 If science funding is to be a goal-directed activity, then evaluations of the science funded cannot normatively privilege values such as objectivity for their own sake. 如果科学资助是一个以目标为导向的活动，那么对受资助的科学研究的评估就不能给予诸如为其自身着想的客观性这样的价值以特权。 ‘Peer Review Can Make Metrics Accountable’ International Workshop on Peer Review Dalian University of Technology Monday, May 21 – Wednesday, May 23, 2012