Building System Models through System Ethnography

1. Building System Models through System Ethnography First Annual Conference on Quantitative Methods & Statistical Applications in Defense Andrew Koehler, PhD Alyson Wilson, PhD Christine Anderson-Cook, PhD Statistical Sciences, D-1 Los Alamos National Laboratory 2/3/2006 LA-UR-06-0951

2. 2 Munitions Stockpile Reliability Assessment - Summary • Goal/Objective: The goal of this project is to develop a dependable and cost-effective suite of statistical methodologies and tools to assess the reliability of weapon stockpiles. • Approach/Tasks • Methodological development • information integration • uncertainty quantification with heterogeneous data • Applications collaboration • Tool development • software for rapid development of systems and statistical models

3. 3 Collaborators and Customers • DoD • MCPD Fallbrook (TOW) • NSWC Corona (RAM, ESSM) • NSWC Yorktown (AMRAAM) • AMCOM/RDEC (Stinger) • DOE • LANL Enhanced Surveillance Campaign • LANL Core Surveillance

4. 4 The fundamental question is how to assess stockpiles as they change over time. • Stockpiles change over time due to materials degradation, life-extension programs, maintenance, use, and other factors. • Assessment requires • the development of system models that capture parts, functions, dynamics, and interactions • the integration of multiple data sources, including historical data, surveillance testing, accelerated life testing, computer model output, and materials characterization.

5. 5 Canister Reliability as Currently Practiced Active Optical Target Detector Control Section Propulsion Section Ordnance Section Guidance Section RAUR= RG*RC*RA* RO* RP*RCst=0.95

6. 6 The (growing) Challenge Suppose that we are trying to assess a stockpile that has • Multiple variants, • Multiple data sources, • Distributed expertise, • Limits on functional testing and that we want • A numerical estimate of current reliability and performance based on individual and group characteristics, • A prediction of how reliability and performance change over time, • Uncertainties on the estimates and predictions, perhaps as part of a capability based surveillance plan design • A system description that captures stockpile environments and use dynamics.

7. 7 Technical Challenges • Facing a multilevel data modeling and inference challenge in order to incorporate system-component surveillance data sources • Keeping track of multi-level data and dependencies • Existing optimal experimental design methods cannot be employed to compare the relative value of multiple, multi-level experiment types

8. 8 Integrating Components of Model into Unified Analysis • To combine the data from these different data sources, we need an approach that allows flexibility: • There is a considerably variability in how much data is observed for different pieces of the system • Not all components will have quality assurance data • The specification limits are thought to be approximations of when the part will fail, but do not necessarily match exactly with the flight data • Observed flight failure modes will not necessarily specify the failure of every component • There is frequently ambiguity about which component failed during flight testing

9. 9 System Ethnography • Capturing hypotheses from all system stakeholders about what components exist in the system, and how those components relate to one another; • Encoding component behaviors as set of rules which can tested against observed system behaviours; • Incorporating dynamic system behaviors across all operational modes of the system; • Linking component state information to quantitative and qualitative data sources; • Performing checks to determine whether component reliability hypotheses are consistent and result in calculable reliability models; • And inferring all possible combinations of component states that can result in observed system behaviors.

10. 10 System Ethnography and Stitching together a System Behavioral Data Model System component logic --missile descriptions and documentation --expert knowledge --existing FMECA --life-cycle/maintenance records

11. 11 System Ethnography and Stitching together a System Behavioral Data Model (II) Missile time- line information

12. 12 Fault/diagnostic/telemetry information

13. 13 Using surveillance information from multiple variants can reduce uncertainty and improve prediction. • We are developing the Graphical Ontology Modeling and Inference Tool (GROMIT) for system representation and qualitative inference. • The gray boxes are parts or functions that appear in other variants of the system.

14. 14 Combine all available information to understand uncertainties in system reliability and performance. • Data is often available from many different experiments: flight tests, component tests, accelerated life tests. • GROMIT allows us to understand what the data tells us about the system. • We also develop statistical methods to formally combine the information into a unified system reliability estimate.

15. 15 GROMIT allows us to combine information from different experts into an integrated system view. • Different subject matter experts understand different parts of the system. • GROMIT highlights potential differences in system assumptions and understanding from various experts, to create a more accurate system representation. • Effective assessment requires an integrated system view.

16. 16 GROMIT captures the in-use dynamics and failure modes of a system. Different failure modes affect the system under various use environments giving more precise information about specific component reliabilities.

17. 17 GROMIT facilitates qualitative exploration of systems. • Any system function or part can be set to any state and the results are propagated throughout the system to produce cut-sets. • For example, if a particular failure mode is observed, we can produce a list of all combinations of parts state which might have caused this. • GROMIT is not binary, but handles multiple states.

18. 18 Stage 1 Stage 8 C1 C2 C3 C28 C29 C30 System Reliability Estimate System What parts are in the system, how specification data links to the parts and… …the reliability information content of a particular system level outcome upon component level performance.

19. 19 Individual Missile Component Information is then Rolled up to Provide System Reliability System Reliability at any age is the product of all of the component reliabilities in a serial system P(system success) = function of component reliabilities

20. 20 Future Directions • “Response Space Knowledge Modeling” • Improved fault isolation • Better characterization of continuous, non-DAG types of dependencies • Stitching together analog FMECA (particularly for very large architectures)