Software Review Process and Level of FAA Involvement

1. Software Review Process and Level of FAA Involvement Software and Complex Electronic Hardware Standardization Conference Brenda S. Ocker, Software Technical Specialist July 27, 2005

2. Presentation Overview • Background • Overview of the Software Review Process • Four Types of Software Reviews • Preparing, Conducting, and Documenting the Software Review • Determining the Level of FAA Involvement • Summary

3. Background • RTCA/DO-178B, Section 9.2 • “Certification authority reviews may take place at the applicant’s facilities or the applicant’s suppliers’ facilities.” • RTCA/DO-178B, Section 10.3 • “The certification authority may review at its discretion the software life cycle processes and their outputs during the software life cycle as discussed in subsection 9.2.”

4. Background • Order 8110.49, Chapter 2 • Provides guidelines for performing software reviews • Documents the review approach, which is detailed in FAA Job Aid “Conducting Software Reviews Prior to Certification”

5. Background • Order 8110.49, Chapter 3 • Provides guidelines for determining the Level of FAA Involvement (LOFI) in a software project • When the FAA should be involved • To what extent the FAA should be involved • Which areas the FAA should focus their involvement

6. Software Review Process Assess LOFI Determine Number, Type, and Timing of Software Reviews ConductSoftware Reviews Use Job Aid & DO-178B

7. Objectives of Software Review • Address technical issues in a timely manner • Physically examine compliance data • Verify adherence to plans and procedures • Monitor designees

8. Review Process & Life Cycles • Begin early in the software life cycle • Integrated throughout the software life cycle • Regular communication between applicant and certification authority

9. Four Types of Software Reviews • Planning • Development • Verification • Final Certification

10. Software Planning Review • Stage of Involvement (SOI) # 1 • Determine if plans and standards provide an acceptable means for satisfying objectives of RTCA/DO-178B

11. Software Planning Review • Conducted when initial software planning process is complete • Plans and standards have been internally reviewed • Plans and standards have been reviewed by SQA • Plans and standards are approved and under configuration control

12. Software Planning Review SOI # 1 Planning Requirements Design • PSAC • SDP • SVP • SCMP • SQAP • Standards • Tool Qual. Plans • Verif. Data • SQA data • SCM data Code/Integration Integration/Test Tables A-1, A-8, A-9, A-10

13. Software Development Review • Stage of Involvement (SOI) # 2 • Determine if software development in accordance with approved plans and standards

14. Software Development Review • Conducted when at least 50% of development data is complete • High-level requirements are documented, reviewed, and traceable to system requirements • Software architecture is defined and reviews and analyses are complete • Low-level requirements are documented, reviewed, and traceable to high-level requirements • Source code implements low-level requirements, is traceable to low-level requirements, and has been reviewed

15. Software Development Review Planning SOI # 2 Requirements Design Code/Integration • Software Req. Data: - High-Level Reqs. - Derived High- Level Reqs. • Req. Standards • Verif. data • SQA data • SCM data Integration/Test • Design Description: - Architecture - Low-Level Reqs. - Derived Low-Level Reqs. • Design Standards • Verif. data • SQA data • SCM data • Source Code • Executable Object Code • Code Standards • Verif. data • SQA data • SCM data Tables A-2, A-3, A-4, A-5, A-8, A-9, A-10

16. Software Verification Review • Stage of Involvement (SOI) # 3 • Determine if software verification in accordance with approved plans • Ensure requirements, design, code and integration appropriately verified • Ensure verification process will achieve • Requirements based test coverage • Appropriate level of structural coverage

17. Software Verification Review • Conducted when at least 50% of verification and testing data is complete • Development data is complete, reviewed, and under configuration control • Test cases and procedures are documented, reviewed, and under configuration control • Test cases and procedures have been executed (formally or informally) • Test results are documented • Testing environment is documented and controlled

18. Software Verification Review Planning Requirements SOI # 3 Design Code/Integration Integration/Test • Test cases and procedures • Test results • Verif. data • SQA data • SCM data Tables A-2, A-6, A-7, A-8, A-9, A-10

19. Final Certification Review • Stage of Involvement (SOI) # 4 • Determine compliance of final product with objectives of RTCA/DO-178B • Verify that all software related problem reports, action items, and certification issues have been addressed

20. Final Certification Review • Conducted when final software build is complete and ready for formal system certification approval • Software Conformity Review has been conducted • Software Accomplishment Summary and Software Configuration Index are complete • All other software life cycle data are complete and under configuration control

21. Final Certification Review Planning Requirements Design • PSAC • SDP • SVP • SCMP • SQAP • Standards • Verif. data • SQA data • SCM data Code/Integration SOI # 4 • Software Req. Data: - High-Level Reqs. - Derived High- Level Reqs. • Verif. data • SQA data • SCM data Integration/Test • Design Description: - Architecture - Low-Level Reqs. - Derived Low-Level Reqs. • Verif. data • SQA data • SCM data • Source Code • Executable Object Code • Verif. data • SQA data • SCM data • Test Cases and Procedures • Test Results • Verif. data • SQA data • SCM data • Cert liaison ALLTables TablesA-1, A-8, A-9, A-10 Tables A-2, A-7 A-8, A-9, A-10 Tables A-2, A-3, A-4, A-5, A-6, A-8, A-9, A-10

22. Review Process • Prepare for the review • Notify the applicant • Conduct the review • Document the results • Brief the applicant • Follow-up activities

23. Preparing for the Review • Assemble review team • Software engineering • Systems engineering • SCM • SQA • Prepare for review • Draft agenda • Coordinate with applicant

24. Notify the Applicant • Provide written notification • Purpose of review • Date and duration • Review team • DO-178B objectives to be addressed and data to be reviewed • Data to be submitted before review • Request appropriate personnel be available to answer questions during review • Recommend applicant perform self-assessment before review

25. Conducting the Review • Certification authority entry briefing • Applicant briefing • Conduct review • Reading • Interviewing • Sampling • Witnessing • Certification authority exit briefing

26. Document the Results • List of all data items reviewed • Explanation of Findings and Observations • Finding: Identification of a failure to show compliance with one or more of the DO-178B objectives • Observation: Identification of potential software life cycle improvement • Identify any other certification issues and action items

27. Follow-up • Provide written report to applicant • Follow-up on findings, observations, certification issues, and action items • Schedule additional reviews as needed

28. Level of FAA Involvement • When the FAA should be involved • To what extent the FAA should be involved • Which areas the FAA should focus their involvement

29. Level of FAA Involvement • Number, type, and depth of reviews will vary depending on: • Software Level • Product attributes (e.g. size, complexity) • New technology or novel design • Experience with DO-178B • Experience with Certification • Designee Support • Other special considerations

30. Determining the LOFI • Applicant and Certification Authority work together to determine LOFI at the beginning of the project • Three Levels of Involvement • High • Medium • Low

31. Determining the LOFI • Software level is the starting point: • Level A = Medium or High • Level B = Medium or High • Level C = Low or Medium • Level D = Low • Reference Table 3-1 in Order 8110.49

32. Determining the LOFI • Need to consider other relevant criteria for levels A, B, and C • Software Certification Experience • Demonstrated Software Development Capability • Software Service History • Current System and Software Application • Designee Capabilities • Reference Table 3-2 in Order 8110.49

33. Figure 3-2: Other Relevant Criteria • Software Certification Experience • Experience with civil aircraft or engine certification • Experience with RTCA/DO-178B • Experience with RTCA/DO-178 or RTCA/DO-178A • Experience with other software standards (other than RTCA/DO-178[ ])

34. Figure 3-2: Other Relevant Criteria • Demonstrated Software Development Capability • Ability to consistently produce RTCA/DO-178B software products • Cooperation, openness, and resource commitments • Ability to manage software development and subcontractors • Capability assessments (for example, Software Engineering Institute Capability Maturity Model, ISO 9001-3) • Development team average based on relevant software development experience

35. Figure 3-2: Other Relevant Criteria • Software Service History • Incidents of software-related problems (as a percentage of affected products) • Company management’s support of DERs • Company software quality assurance organization and configuration management process • Company stability and commitment to safety • Success of past company certification efforts

36. Figure 3-2: Other Relevant Criteria • Current System and Software Application • Complexity of the system architecture, functions, and interfaces. • Complexity and size of the software and safety features • Novelty of design and use of new technology • Software development and verification environment • Use of alternative methods or additional considerations

37. Figure 3-2: Other Relevant Criteria • Designee Capabilities • Experience of DER(s) with RTCA/DO-178B • Designee authority, autonomy, and independence • Designee cooperation, openness, and issue resolution effectiveness • Relevance of assigned DERs experience. • Designees’ current workload. • Experience of DER(s) with other software standards (other than RTCA/DO-178[])

38. Figure 3-2: Other Relevant Criteria • Minimum and Maximum values for each criteria • “Experience with RTCA/DO-178B” • No projects = 0 • 2-4 projects = 5 • More than 5 projects = 10 • “Experience with other software standards” • No projects = 0 • 4-6 projects = 2 • More than 7 projects = 4 • Scale is weighted • “Experience with RTCA/DO-178B” is weighted higher than “Experience with other software standards”

39. Total Score Result (TSR) (from figure 3-2) Software Level A Software Level B Software Level C Software Level D TSR < 80 HIGH HIGH MEDIUM LOW 130 < TSR MEDIUM MEDIUM LOW LOW 80 < TSR < 130 HIGH MEDIUM MEDIUM LOW Figure 3-3: LOFI Determination NOTE 1: If the TSR is close to the TSR boundary values (that is, 80or 130), use the software level and engineering judgment to determine the most appropriate LOFI. NOTE 2: If any criterion in figure 3-2 is not applicable, the assessormay use the average value or adjust the figure 3-3 boundaries.

40. Determining the LOFI • Projects with issues that require new FAA policy typically require more FAA involvement • Level A and B = High • Level C and D = Medium

41. Determining the LOFI • Document specifics of FAA involvement using Appendix 1 in Order 8110.49 • Number and type of FAA on-site reviews • Number and type of FAA desk reviews • Data to be submitted to the FAA • Delegation to DERs

42. Appendix 1: LOFI Worksheet Plan Based on LOFI Assessment: (for example, number of FAA on-site reviews, number of FAA desk reviews, data to be submitted to the FAA, and delegation to DERs) Mid-Mid-Project Adjustments: (based on project improvements or problems) Actual Project Results: (for example, number of FAA on-site reviews, number of FAA desk reviews, data submitted to the FAA, and delegation to DERs)

43. Example: High LOFI • Minimal delegation • Recommend approval of all data • Software specialists involved • FAA involvement throughout software life cycle • At least 2 on-site reviews • Submittal of all software plans • Submittal of SAS, SCI, Verification Results, and Compliance Matrix

44. Example: Medium LOFI • Moderate delegation • Recommend Approval of PSAC and SAS • Approve SCI and other data • FAA involvement at beginning and end of software life cycle • Software specialists may be involved • At least 1 on-site review • Submittal of PSAC, SCI, and SAS • Potential submittal of other software plans and Compliance Matrix

45. Example: Low LOFI • Maximum delegation • DER Recommend Approval of PSAC and SAS • DER Approve all other software data • Minimal FAA involvement • No on site reviews • Few desk-top reviews • Software specialists not involved • Submittal of PSAC, SCI, SAS

46. Additional Information • Reference the FAA Software Job Aid "Conducting Software Reviews Prior to Certification” • View “Using the Software Job-Aid to Conduct Software Reviews” Video • Available through FAA Web-site:http://www.faa.gov/certification/aircraft/ • Link to “Aircraft Certification Software” on lower right hand side of page

47. Summary • Overview of the Software Review Process • Four Types of Software Reviews • Planning • Development • Verification • Final • Preparing, Conducting, and Documenting the Software Review

48. Summary • Determine the LOFI at beginning of project • Order 8110.49, Figures 3-1, 3-2, and 3-3 • Document LOFI Assessment • Order 8110.49, Appendix 1 • Make adjustments to the LOFI during the project, if needed • Reference the examples • Order 8110.49, Appendix 2, 3, and 4