Dr. Sergio Guarro Distinguished Engineer, The Aerospace Corporation

1. Logic -Quantitative Framework for Decisionmaker's Management of Mission Failure Risk USC – CSSE Annual Research Review Workshop7 March 2012 Dr. Sergio Guarro Distinguished Engineer, The Aerospace Corporation

2. Background and Context of The Aerospace Corporation Mission Assurance and Risk Framework • Space missions are unforgiving • The rule of the game is essentially “one strike and you are out” – i.e., minimal possibility exists for remedying problems during mission execution • Volume and mass constraints also limit the amount of redundancy that can be used as insurance against failures • The possibility of failure must be understood and managed at the lowest levels of system design detail • Because of the above Mission Assurance and Mission Risk Assessment processes are given great attention and priority in the range of activities our company executes on behalf of our U.S. Government space program Customers • Comprehensive Program Offices’ mission assurance task plans and assessment processes • Special issues addressed with specialized Engineering & Technology Group support • Aerospace specialists’ analyses in support of Customers’ independent review team assessments

3. APR / ASMR Framework & Process • The Aerospace Corporation (“Aerospace”) President Review / Senior Management Review (APR / ASMR) process is the concluding synthesis of a full cycle of assurance and risk assessment applied to supported National Security Space (NSS) programs, to provide the decision-maker with the analytical means to judge and manage the risk of mission failure • Structured integration of Risk Assessment (RA) and Management (RM) information produced by program contractor(s) and Government / Aerospace Program Office is key to success of APR/ASMR process • The logic – quantitative risk framework presented here is the result of the most recent development to provide reference guidance for the APR / ASMR risk assessment processes • The guidance is documented in a corporate Technical Instruction published by The Aerospace Corporation Corporate Chief Engineer Office and supported by more detailed documentation produced by The Aerospace Corporation Systems Engineering Division APR / ASMR RA Process Focus on Mission Risk Only Address both Programmatic &Mission Risk Indpdt. Program Reviews Indpdt. Review Team RA Process PO RM Process Contractor RM Process

4. Objectives and Flow of Logic-Quantitative Risk Framework • Key objectives • Clear identification of key factors and events that can determine a mission impact • Assessment of risk in objective probability and mission-impact dimensions • Avoid qualitative definition of likelihood and consequences that are intrinsically subject to different interpretations by different audiences • Separation of risk definition and assessment from decision process • Value judgment of risk is the decision-maker’s, not the assessor’s, responsibility • Typical execution flow Program Office Risk Screening MA Plan & Scope Risk Communication& DecisionSupport Risk Assessment & Rating Risk Scenario Definition Preliminary Identification & Evaluation Eng. &Tech. Group

5. Risk Identification Using Mission Assurance Baseline Risk identification proceeds from the basic concept of risk as deviation from “mission assurance baseline” (MAB): Each space system mission item (SSMI) within the assessment scope is evaluated from this perspective A potential SSMI risk item is identified as a significant deviation from the desired level of quality in a set of reference mission assurance attributes Guidance document s define the set of attributes to be evaluated and the severity criteria to determine whether any existing deviations are significant enough to call for the formal definition of an associated risk This risk identification concept is the application of a general concept that relates risk directly to the Aerospace MA (Mission Assurance) processes

6. 10-20 Mission Impacting Major Technical Risks Risk Item Screening Technical risks • Apply filter to preliminarily-identified potential risks • Apply full assessment and quantification technique to mission impacting major technical risks yes Define & Assess Risk Scenario yes yes Preliminary Risk Identification Significant Deviation from Baseline? Mission Impact? Technical Impact? 100s of non-quantitative potential risks preliminarily identified (e.g., in MA Baseline task executions) no no no Lower Level Issues Cost & Schedule Risks Lower Severity Risks

7. Risk Scenario Definition • A risk scenario is defined to initiate the analysis / assessment portion of the process for post-screen mission risks Definition: A Risk Scenariois a system or mission condition that can be formally described as a cause-effect sequence of events the occurrence of which may cause a mission risk impact and associated consequences to be realized. • The reference risk scenario identifies in logic event sequence diagram (ESD) format the key chance events / conditions that may affect the outcome of a given risk in terms of probability and consequence severity • This may include risk control measures expressly introduced by a program to counter an identified risk: • preventive control measures (PCMs),when executed successfully, eliminate altogether the potential mission impact of a given risk • mitigative control measures (MCMs), when executed successfully, reduce the potential mission impact of a given risk by some predicted amount that can be quantified as a consequence reduction factor SCENARIO EVENT SEQUENCE DIAGRAM (ESD) - including PCM & MCM events MCMs are presentand successful PCMs are present and successful Unmitigated Mission Impact is realized Initiating Event occurs yes yes yes no no Intermediate Event X occurs Intermediate Event Y occurs no no no yes yes Mitigated Mission Impact is realized No Mission Impact is realized

8. Risk Rating Once a reference risk scenario has been defined and expressed in standard ESD form risk can be assessed by estimating: Likelihood / probability of initiating event Conditional likelihood / probability of intermediate events Including probability of success of PCMs and MCMs Severity / magnitude of mission performance shortfall resulting from any mission impact scenario outcomes Performance shortfall reduction factors associated with MCM-event successful outcomes The guidance documentation provides the simple formulations by which risk scenarios can be quantified and rated in summary “probability of consequence severity” form, using the above ESD quantification parameters

9. When a risk involves consequences in multiple mission performance dimensions a combined Mission Shortfall Metric (MSM) needs to be developed This can be done by mapping hypothetical shortfall magnitudes relative to individual key performance parameters into a single MSM scale, i.e., essentially defining a simple “mission utility function” (in the potential shortfall direction) Rating of Multiple Performance Consequence Effects 0 % Image Resolution Shortfall 10 % 20% 30% 40% Legend Iso-consequence calibration line Performance Requirement Performance Parameter Shortfall (% of required value) Mission Shortfall Metric (MSM) 1 No Missn. Value 0 Full Missn. Value 0.25 0.50 0.75 Data Rate Shortfall 20 % 40% 60% 80% 0 %

10. Risk Communication and Decision Support The recommended format of risk communication is a “probability vs. consequence severity” risk map on which appropriate areas of risk have been pre-identified for reference according to decision makers’ input and directives Uncertainty in both probability and consequence magnitude is also displayed Estimates of individual Mission Riskswith significant uncertainty Estimates of individual Mission Riskswith low uncertainty

11. Use in Decision-Making: Power Distribution Shorts Scenario Example • Risks flow from initiator through intermediate events to impacts • The initiator is not the risk • Include and show “delta effect” of any preventive or mitigative control measures (PCMs, MCMs) • Benefits: • Easier to understand and more thorough “risk statement” • Clearly identifies key events and factors, which strongly influence risk outcome • Shows effectiveness of prevention and mitigation • Makes it easier to resolve disputes Short Load > 20 Amps Mission Loss Solar Panel Wiring Insulation Cracked / Frayed Short Occurs Short Is in Unprotected Section Short Load 6 to 20 Amps Mission Degradation Short Load < 6 Amps No / Minor Impact PCM1 Add ExtraInsulation PCM2 Add DiodeProtection MCM1 Sectorize Solar Panel Potential Controls

12. Power Shorts Scenario Example Details

13. Power Shorts Scenario Results • Assessment results suggested that some risk control measures would be warranted, if their introduction were technically feasible

14. Defined and formulated to support Decision-makers’ assessment and management of risk of mission failure Clear, unequivocal definition / description of all “selected risks” “Reference Scenario” Format Distinction between assessment, display/communication, and decision-support aspects of risk process Assessment via objective, quantifiable metrics Quantification recommended for objectivity, not to project impression of precision Strong recommendation to explicitly display assessment uncertainty In Summary: Key Points of Logic-Quantitative Risk Framework

15. Backup Charts

16. APR MISSION - ASSET BASELINE ATTRIBUTES "PROCESS" / "PRODUCT" BASELINE DEFINITION DEVIATION FROM BASELINE NATURE of D EVIATION Modest Significant Large Process Product 1 Design Assurance Factors 1.1 Residual issues with SSMI design / engineering and No residual issues X interface specifications 2 Manufacturing Assurance Factors 2.1 Indication of SSMI manufacturing technology issues No indication of issues X 2.2 Deviations from of SSMI manufacturing quality control No deviations X processes 3 IT&E Factors 3.1 SSMI TLYF exceptions No unassessed / unjustified X exceptions 3.2 IT&E process deviations from SSMI requirement No deviations X verification objectives 3.3 SSMI requirements, including interface and reliability No IT&E results deviations from X requirements, not verified by results of IT&E process requirements (test, analysis, or demonstration) 4 Operations Readiness Factors 4.1 Product evidence of SSMI integration and mission No evidence of issues X readiness issues 4.2 Deviations from SSMI anomaly resolution plans No deviations X 4.3 Residual liens against SSMI on Orbit Testing and No residual liens X Operations Certification requirements 5 MA Disciplines Factors 5.1 Deviations from Specifications and Standards No deviations X requirements applicable to SSMI 6 Other Factors 6.1 SSMI - specific issue #1 (describe) No issue TBD … … … 6.2 SSMI - specific issue #N (describe) No issue TBD Example of MA Baseline Attributes for Risk Identification

17. Initiating Event Identification in Risk Scenario ESD SSMI BASELINE FACTORS TO BE EXAMINED TO DEFINE RISK SCENARIO INITIAL CONDITION • The initiating event in a risk scenario is identified according to the nature of the baseline deviation(s) initially identifying the risk Requirements Deviations ? Design Deviations ? Manufacturing & Assembly Deviations ? MA Discipline Specs & Stds Deviations? IT & E Deviations ? Operational Readiness Deviations ? If evidence of deviations exist, is it in process or product attributes ? If any deviations exist, is their magnitude moderate, significant, or large (M, S, or L) ? RISK-SCENARIO INITIAL CONDITION DEFINED IN TERMS OF ANSWERS TO ABOVE QUESTIONS

18. Examples of ESD Templates Provided in Risk GuidanceDocument ESD Template for Risk Scenario Driven by SSMI Product Attribute Deviation SSMI productexhibits [ M / S / L ] deviation from[req./des./… ] baseline Is SSMI deviationfully controlledby PCMs added after risk identification ? Is SSMI deviationfully controlledby built-in system design features (e.g., redundancy, operational options, etc.) ? Is SSMI deviationmitigated by MCMs added after risk identification ? Unmitigated Mission Shortfalls are realized no no no Less severe Mission Shortfalls are realized ESD Template for Risk Scenario Driven by SSMI Process Attribute Deviation yes yes yes SSMI processexhibits deviation frombaseline Is SSMI product deviation“moderate” (M)? Is a SSMI productdeviation frombaseline producedas a result ? Enter productdeviation ESD w/ “M” deviationcondition yes yes No Mission Shortfalls are realized Is SSMI product deviation“significant” (S)? Enter “productdeviation” ESD w/ “S” deviationcondition no no yes Enter “productdeviation” ESD w/ “l” deviationcondition no No Mission Shortfalls are realized