1 / 9

Thoughts on how to improve reviews

Thoughts on how to improve reviews. Paul Francis (Presented by Robbert Van Renesse) Cornell. Main problem with current system. Reviews are often of poor quality I suspect main reason is that PC members are overworked Both ideas in this presentation are focused on this one problem alone.

aragsdale
Télécharger la présentation

Thoughts on how to improve reviews

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Thoughts on how to improve reviews Paul Francis (Presented by Robbert Van Renesse) Cornell

  2. Main problem with current system • Reviews are often of poor quality • I suspect main reason is that PC members are overworked • Both ideas in this presentation are focused on this one problem alone

  3. Idea One: • Allow PC members to ask authors simple “where in the paper can I find this?” questions • Authors can answer ONLY with page, column, and line numbers • Why? • Often when I reject a paper, it is based on one or a few specific flaws. It would be good to verify that I’m not overlooking something.

  4. A reviewer comment • “But you can do this today...just ask the PC chairs to forward an email” • My rebuttal: • Bothering the PC chairs is a significant deterrent • Should really limit author reply to text already in paper...otherwise author is effectively submitting more than what is in the paper

  5. Today

  6. Proposal One: Pass reviews on

  7. Observations • Reviewer gets the benefit of previous reviewers • Various ways this can be exploited: • To improve review: Reviewer first does complete unbiased review, then uses previous reviews as sanity check • To reduce work: Reviewer scans paper, previous reviews, and rebuttal, and then does only enough work to decide if author has overcome previous criticisms • This can be done in good faith

  8. Observations • Good paper should be able to overcome earlier criticisms • Without having to torque paper to satisfy future reviewers.... • Instead author can use rebuttal • Truly bad paper should not be able to overcome earlier criticisms • Prevents bad paper from retrying until it gets lucky • Might discourage “hail-mary” submissions

  9. A reviewer comment • “This will promote reviewer laziness” • My rebuttal: • A good-faith reviewer will benefit from earlier reviews • A lazy reviewer already has many ways of being lazy (form instant opinion, do minimal work to support opinion...)

More Related