TALES FROM THE TRENCHES

1. TALES FROM THE TRENCHES • TASK OUTLINE

2. INSPECTIONS APPLIED TO A TEST DEVELOPMENT PROJECT • Inspections were conducted to support Operational Flight Program (OFP) Test Script Development for the Avionics Integration Support Facility (AISF) • AISF is an OFP software development and test facility • Supports the development and maintenance of several of Avionics Operational Flight Programs (OFPs) • OFPs had been previously developed by various Suppliers • OFP Software test procedures were required to be automated and consistent with each other to reduce manpower levels • Existing procedures varied from highly automated to entirely manual • Test environments varied from Supplier to Supplier • Availability of Supplier test bench documentation was limited • Existing OFP test procedures were converted into “OFP test scripts” • “Test scripts” were written using a high level BASIC-like language

3. TEST SCRIPT DEVELOPMENT • Script Development Risk Factors • Size of Procedure • Level of Pre-existing Automation • Level of Supplier Documentation • Resident Expertise • Level of Expertise of Script Developer • Script Development Methodology • Analysis of conversion task • Identification of “Representative Test Scripts” • Conversion of “Representative Test Scripts” • Walkthrough of “Representative Test Scripts” • Conversion of remaining test scripts • Integration utilizing the “Representative Test Scripts” • Debug and test of remaining test scripts

4. CONVERSION WALKTHROUGH • Two Phases • Internal Review (Informal with I&T members and Systems) • Formal Review (With Systems, I&T and Software) • Topics Discussed in Internal Review • Representative samples of scripts using available syntax • Identify “unknowns” • Topics Discussed in Formal Review • Samples of scripts, which fully cover all box functions, converted to ATU • Conversion Approach • Manual or translator to be developed • Pro/Cons if multiple approaches can be pursued, including schedule impacts • Risk areas • Metrics • Schedules

5. TEST CONVERSIONWALKTHROUGH PROCESS Identify “Representative” Test Cases Develop “Top-Down” Functional Flows for Test Cases Develop ATL for Functions/ Utilities Conduct Informal Review with I&T and Systems Representative Revise “Top-Down” Structure/ Utilities Revise ATL for Selected Test Cases Conduct “Formal” Review with Management, Software, Systems, and I&T Obtain Approval on Test Conversion Approach Fully Develop “Representative” Test Cases and Deliver to Software for Checkout Checkout on Bench Software/Script Revisions Continue to Develop Remainder of Test Cases Verification of OFP Qual Capability

6. WALKTHROUGH CHECKLIST • Overview Of Existing Vendor Test • Mapping Of Existing Vendor Test to AISF Test • Overview Of AISF Test, Including Names Of FTDS and Test Cases • User Interface (E.G. TUI Screen For Bench Configuration) - Formal Walkthrough Only • Simple Block Diagram (CTC, ITC, Emulator, Test Equipment) - Formal Walkthrough Only • Box Input/Output Identified by AISF S/W Variable Name or HEX Overwrite Syntax, depending on the implementation • Calling Trees for Script Procedures and Utilities for each FTD • General types of Inputs and Outputs, as well as Functional Descriptions of Script Procedures and Utilities

7. WALKTHROUGH CHECKLIST (Con’t) • Traceability Matrix, with selected “Representative” Test Cases Bolded, and Bolded “X” for the “Representative” Function • Scripted “Representative” Test Cases using AISF ATL • Total Lines Of Estimated Script (Include Comments In Count) • Conversion Plan, Identifying Manual Conversion or Translator. If using a Translator, Identify Translator Development Plan and Translator Processing Requirements

8. TEST SCRIPT INTEGRATION • Bench Integration and Risk Reduction of Test Script Development Effort focused around the integration and debug of “Representative Test Cases” • Integration Risk Factors • Number of Bench Functions • Experience Level of the Test Engineer(s) • Bench Integration order • In House Expertise • Overall Project Integration Effort • 430 K SLOS of test scripts • 1 M SLOC of simulation and support software • Eight OFPs • Eight LRU Benches

9. IMPACT OF WALKTHROUGHS ON TEST SCRIPT INTEGRATION SUCCESS • Walkthrough Success Factors • Peer Review vs. Expert Review • Preparation of reviewers • Action Items Tracked and Closed • Completeness of Review • Project Completion Impacts • Budget Impacts • Schedule Impacts • Rework • Work-Arounds • Completion Criteria

10. IMPACT OF WALKTHROUGHS ON TEST SCRIPT INTEGRATION SUCCESS (Con’t) • Premise: If Walkthroughs had no impact, then the Risk Factors for Integration would correlate with the Integration Success (...assuming all things being equal) • Conclusion: Successful Walkthroughs reduced integration risk and Project Completion Impacts

11. LESSONS LEARNED • All Engineers should be trained in Walkthroughs and Peer Review Process • QA participation is crucial • CM participation is crucial • “Non-software” should be handled consistent with Software Best Practices • New engineers out of school are less resistant to Process Change • Make sure management, team leads and backups have “bought into” process • Don’t slack up on the later activities • Do inspections on all aspects of the development process • Don’t cut corners, even if you think you know what you are doing • Handle all elements of the system as if you are reviewing software • Reduces interface risk

12. BOTTOM LINE • SEI Processes reduce risk when applied to • Systems Level Integration and Test Activities