1 / 14

Project LEAP: Addressing Measurement Dysfunction in Review

The LEAP project addresses the pervasive issue of Measurement Dysfunction in Formal Technical Reviews (FTRs). Traditional methods can lead to counterproductive outcomes, distorting the metrics meant to improve review quality. LEAP introduces an alternative approach that focuses on the individual reviewer, enabling personalized insights without compromising privacy. By facilitating the collection and analysis of essential data, LEAP enhances defect detection and overall review effectiveness. This presentation covers the problems, outcomes, and future directions of LEAP's implementation in the software development domain.

hayes
Télécharger la présentation

Project LEAP: Addressing Measurement Dysfunction in Review

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Project LEAP: Addressing Measurement Dysfunction in Review Carleton Moore Collaborative Software Development Laboratory Department of Information & Computer Sciences University of Hawaii http://csdl.ics.hawaii.edu/

  2. Outline • Problem • Measurement Dysfunction • An alternative Approach: LEAP • Evaluation • Results • Future Directions • Conclusions

  3. Problem • Trying to improve Formal Technical Review (FTR) quality often leads to Measurement Dysfunction. • Collect metrics on the FTR practice to help improve future FTRs. Modify FTR Practice Evaluate Metrics FTR Metrics

  4. Measurement Dysfunction • When the act of measurement affects the organization in a counter-productive fashion, which leads to results directly counter to those intended by the organization for the measurement. Robert Austin, “Measuring and Managing Performance in Organizations” • In essence, “metrics that backfire”. • Example: Russian Boot Factory

  5. Measurement Dysfunction inFTR • Goal: Increase the number of “important” defects found • Defect severity inflation/deflation • Goal: Improve FTR defect detection • Defect density inflation • Goal: Reviewers prepared for review • Reviewer preparation time inflation • Goal: Improve defect detection rate • Defect discovery inflation

  6. The Alternative: LEAP • Focus on individual reviewers • Better reviewers => Better reviews • Tools and processes must obey LEAP constraints: • Lightweight, few process constraints • Empirical, both qualitative and quantitative • Anti-measurement dysfunction • Personal, sensitive data is private • http://csdl.ics.hawaii.edu/Tools/LEAP/LEAP.html

  7. LEAP: The Personal Perspective • (Semi or totally) automated recording of: • Time spent on review activities • Defects/Issues resulting from activities • Work product characteristics (size, type, etc.) • Provides accessible, just-in-time analyses of: • Effort, defects, work product characteristics • Which leads to: • Personal insights, documented as checklists

  8. Groups of reviewers share insights: Defects they find in each other’s work via review. Checklists they have developed based upon their private data. But, they do not share measurements! LEAP: The Group Perspective

  9. Essential LEAP Services • Data collection • In-process time, defects,checklists • Data analysis • Trends, frequencies, relevance • Data distribution • Email, web

  10. Evaluation • Use LEAP toolkit for reviews • Internal CSDL use of LEAP for review • Industrial adoption • Leap Data Obfuscater & Web Site • Obfuscate identifying information in Leap data • Publish • Defects • Checklists • Patterns

  11. Results • Internal CSDL usage: • Code reviews • Technical report reviews • LEAP tool kit defect reporting • No effort data exchanged • Just Defects and Checklists • LEAP used and evaluated by over 5 organizations

  12. More Results • Implemented a LEAP data obfuscater • LEAP used in 2 software engineering classes

  13. Future Directions • Industry & academic adoption of Leap for review and process improvement • Investigation of Reviews using Leap • Online repository of Leap data • Common defects • Checklists • Patterns

  14. Conclusions • FTR is subject to Measurement Dysfunction • Focusing on individual reviewers can improve FTR • LEAP helps improve reviewers and reduce Measurement Dysfunction

More Related