Formal Technical Reviews Matt Graham 18 November 2004 EECS 814 University of Kansas
Introduction Generic Inspection and Variations Inspection throughout Development Cycle FTR Metrics Tools Conclusion Outline
FTR: What it is... ::Input:: • A method involving a structured encounter in which a group of technical personnel analyzes an artifact according to a well-defined process. ::Output:: • A structured artifact that assesses or improves the quality of the artifact as well as the quality of the method.
A structured FTR is a forum for: FTR: What it is... • Finding defect information for the Author. • Educating peers about product. • Providing fault likelihood data for testers. • Providing detailed status report for managers. • Allowing process improvement group a test to measure.
Walk-through No Measurements Often informal, impromptu Main purpose is developer training FTR: What it isn’t...
Why Review? • Improves overall product quality • Early detection of defects • Reduces rework and its associated costs • Educate the participants and provide training • Setting standards of excellence / maintain process improvement momentum • Improve schedule performance • IBM reported 83% and AT&T 92% defect detection through inspections
Inspection Process Overview Planning • 6 Steps • Team • 2 - 6 members • Experienced and involved in product development • Distinct and important roles Overview Defect Detection Defect Collection Defect Correction Follow-Up
Inspection Process Overview Planning • Roles: • Organizer – plans the inspection activities • Moderator – ensures that procedures are followed and moderates the meetings. • Inspectors – responsible for detecting defects in the product • Presenter – presents the product in logical fashion paraphrased at a suitable rate • Author – author of the developed product. • Recorder – records the defects during the meeting • Collector – collects the defects if there is no meeting. Overview Defect Detection Defect Collection Defect Correction Follow-Up
Planning Planning • Objective – organize inspection • Check to see if work products pass the entry criteria • Select inspection participants and assign roles • Schedule inspection meeting • Distribute inspection material • Prepare inspection documentation • Ensure product inspection readiness Overview Defect Detection Defect Collection Defect Correction Follow-Up
Objective Planning • Objective – educate participants on the product being inspected • Author explains the inspection materials • Can be beneficial when: • The inspection artifact is complex • If the artifact is part of large system • If new members participate in inspection • Could be rolled into previous or next phase if product is simple enough • Communicate inspection goals! Overview Defect Detection Defect Collection Defect Correction Follow-Up
Defect Detection Planning • Objective – to find defects • Can be divided into 2 phases: • Preparation (individual) phase. • Meeting (group) phases. • Individual preparation with the goal of detecting defects can help inspectors to be more prepared. Overview Defect Detection Defect Collection Defect Correction Follow-Up
Defect Collection Planning • Objective – collectively agree and document the defects (triage) • Decide on further inspection needed or not • Decisions made subjectively as a group. Overview Defect Detection Defect Collection Defect Correction Follow-Up
Defect Collection Planning • Objective – correct the defects collected in the previous phases • The author makes the detected changes and sends it for follow-up Overview Defect Detection Defect Collection Defect Correction Follow-Up
Follow Up Planning • Objective – to make sure all defects are resolved • Only one person involved and he/she verifies the defect resolution • Verifier should “return and report” to inspection team Overview Defect Detection Defect Collection Defect Correction Follow-Up
Formal Inspection Methods • Fagan Inspection • Asynchronous Inspection • Phased Inspections • N-Fold Inspections
Fagan Inspection Overview • 4 Roles • Author • Reader • Moderator • Scribe • Meeting centric – cost of scheduling and time • Meetings add little to defect detection Preparation Inspection Rework Follow-Up
Asynchronous Software Inspection Initialization • Team • Author not involved • Moderator and Inspectors Inspection Review Inspection Compile Final Defect List Re-Work Re-Work
Asynchronous Software Inspection Initialization • No meetings • Easy to assess participation • Process improvement from documenting all correspondences • Allows parallel communication • Can be distributed in space and time • Eliminates group approval Inspection Review Inspection Compile Final Defect List Re-Work Re-Work
Asynchronous Software Inspection Initialization • Moderator sends out material • Initial Individual Review – create list of defects • Circulate the copy of defect list to all inspectors and discuss via email • Individual Review – update defect list and send to Moderator • Moderator compiles final defect list, send it to author and follow up – eliminates group approval Inspection Review Inspection Compile Final Defect List Re-Work Re-Work
Phased Inspections • Consists of several coordinated partial inspections called phases • Each phase inspects for a specific property or small set of related properties • Each phase responsible for thorough checking of the properties • Some phases have single inspectors and others have multiple inspectors • Reference documents are provided to inspectors at the beginning of each phase and each inspects using checklists • Domain specific checklists • Application specific checklists
N-fold Inspection • N independent teams inspect the same product using traditional inspection method • Collective effort of multiple teams more faults than a single team • Moderator collects faults from independent teams and composes the final defect list
Requirements Inspection • Reading Techniques are techniques to analyze work product during defect detection • Ad-hoc • Checklist based reading • Scenario based readings • Perspective based reading • Defect based reading
Perspective based Reading • Inspectors stand in for specific stakeholders in the document to verify the quality of requirements. • Designer – detail for creating system components • Tester – detail for constructing test plans • User – correct functionality • For each perspective, reviewer creates high level work products from the requirements document
Code Inspections • Cognitive models for program comprehension • Bottom-up • Top-Down • Integrated • Systematic and as-needed comprehension
FTR Metrics • Helps you measure the effectiveness of the inspection • Aids in continuous process improvement • Provides feedback to management • A unit of work product varies often: • Usually 1000 lines of source code (not including comments) • Page counts and line counts for text
Some Generic Metrics • Average preparation effort and preparation rate per unit of material • Average examination effort per unit of material • Average explanation rate per unit of material • Average number of defects and major defects found per unit of material • Average hours per defect and per major defect
Nine Key Metrics by AT&T for Code Inspections • Total Non Comment lines of source code inspected, in KLOC • Average lines of code inspected • Average preparation rate • Average inspection rate • Average effort per KLOC • Average effort per fault detected. • Average faults detected per KLOC • Percentage of re-inspections • Defect-removal efficiency
Tools • Various tools for different inspection methods. • ICICLE – for inspection of C & C++ programs • Scrutiny & InspeQ for specific inspection processes • ASSIST –supports generic inspection process
Choosing a tool... • Threads of discussion • Sharing of Information • Train of thought • Visual Cues • Reaching a consensus • Coordination
The Future of FTR • FTR adoption is slow: • Managers see added cost of inspections instead of the benefits of greatly reduced defect leakage • Practitioners already have tight schedules and reluctant to take on additional responsibilities • Companies postpone adoption since peer reviews are part of CMM level 3.
FTR Barriers • Insufficient preparation • Moderator domination • Incorrect review rate • Ego-involvement and personality conflict • Issue resolution and meeting digression • Recording difficulties and clerical overhead
Conclusion • Organization’s size, culture and industry should be considered in deciding on the FTR method to use. • Review does not replace testing but can make it easier • Success of inspection lies in: • Process Adherence • Systematic Reading technique • Optimization
Philip Johnson: http://www2.ics.hawaii.edu/~johnson/FTR/ Tom Gilb: http://www.result-planning.com/ IEEE STD 1024-1197 FTR: References