1 / 24

Strategy for quality measurement

Strategy for quality measurement. Richard Nakamura. PhD CSR Advisory Council May 2014. In most production systems, there is a speed, cost, quality trade-off, but this is not a zero-sum game. We have just discussed speed and to some extent, cost. Effectiveness Review quality.

shad-pace
Télécharger la présentation

Strategy for quality measurement

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Strategy for quality measurement Richard Nakamura. PhD CSR Advisory Council May 2014

  2. In most production systems, there is a speed, cost, quality trade-off, but this is not a zero-sum game. We have just discussed speed and to some extent, cost.

  3. EffectivenessReview quality

  4. For Problem A, we have focused on:Attracting:…the best reviewers …the best Chairs …the best SROs

  5. After we put together the best review committees, they must work in the best platform/format to determine merit of applications

  6. Best reviewers -…recruit mostly senior scientists …funded awardees of NIH…w/ strong publications…positive review experience …respect from other reviewers…and diversity that aids the SRG

  7. To get the best reviewers, we offer -…continuous submission…new forms of attendance…meetings on West coast

  8. But -…we take away coffee…we make travel less convenient…we issue 1099 forms for all payments

  9. Today, NIH wants to know how to measure quality so that we can systematically improve it.

  10. To the extent that we have a standard, it has been to examine the results of grants ten years after award as evaluated by senior scientists. This has been done for the Pioneer awards (for 7 years) but provides a very slow feedback loop. Can we do better?

  11. We would like to have measures available more immediately than 10 years after award that predict the longer outcome.

  12. How do we get there?To begin with we need to have estimates of the reliability of review scoring (Rene Etcheberrigaray), so we can power our other measures of quality and interventions correctly.

  13. We need evidence that our scoring approaches are not inappropriately biased – Monica Basco.

  14. In measuring quality, CSR takes seriously the opinions of our scientific reviewers: Surveys – CSR quick feedback (Mary Ann Guadagno)Committee ranking within IRGS (Problem B and last Council)

  15. Is the application ranking system working? (Amy Rubinstein)Do our committees favor conformity? (Seymour Garte)

  16. What other approaches could help us evaluate quality? (actively pursued by other groups and not covered today)Bibliometrics (DPCPSI and CSR): Citations at individual, group levels Relative citation rating; h-index Concerns about citation manipulation

  17. Review committee assignments and coverageNetwork analysis - Application and publication analysis of applicants and reviewersCreation of computer assisted assignment

  18. AARR BBBP BCMB BDCN BST CB CVRS DKUS EMNR ETTN GGG HDM A System Level Representation of Scientific Interests IDM IFCN IMM MDCN MOSS OBT OTC PSE RPHB SBIB VH Other

  19. Biological Chemistry & Macromolecular Biophysics

  20. Infectious Diseases and Microbiology

  21. Questions? Comments?CSRDirector@csr.nih.gov

More Related