Data Chapter 9 Stuart Starr MITRE

1. Prepare for Success NATO Code of Best Practice (COBP) for C2 Assessment 3 Problem Formulation Sponsor Problem 4 Solution Strategy 6 Human & Organisational Issues 5 Measures of Merit (MoM) 7 Data Chapter 9 Stuart Starr MITRE Scenarios 8 Methods & Tools Products 11 9 Data 10 Assess Risk

2. Data Prepare for Success 3 Problem Problem Formulation Formulation Sponsor Sponsor Problem Problem 4 Solution Solution Strategy Strategy 6 Human & Human & Organizational Organizational Issues Issues 5 Measures of Measures of 7 Scenarios Scenarios Merit ( MoM ) Merit ( MoM ) 8 Methods Models & Tools & Tools 11 9 Products Products Data Data 10 Assess Assess Risk Risk

3. Agenda • Definitions, Taxonomy • Nature of the Problem • Key Needs, Best Practices • Summary

4. Definitions • “Data are factual information that are organized for analysis and in a form suitable for machine processing” • “Metadata are information about information; e.g., • Documentation of the attributes of data directly attached to the data • Characterization about reliability of the data”

5. Data Taxonomy

6. Nature of the Data Problem (1 of 2) “The government is very keen on amassing statistics. They collect them, add them, raise them to the n-th power, take the cube root and prepare wonderful diagrams. But you must never forget that every one of these figures comes in the first instance from the village watchman, who just puts down what he damn pleases.” -- Comment of an English judge on the subject of Indian statistics; Quoted in Sir Josiah Stamp in “Some Economic Matters in Modern Life”

7. Nature of the Data Problem (2 of 2) • “Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.” Sherlock Holmes • “Metadata! Metadata! Metadata!” Simone Youngblood, MORS Workshop • “Theory without data = philosophy; data without theory = noise” Anonymous

8. Selected Data Problems The data problems that the C2 assessment community are facing are not unique to it -- many other communities seem to have the same issues, including, inter alia, • Data acquisition • Data conversion • Data sharing • Data reuse • Data purity • Metadata policy • Data shelf life • Data naming conventions • Data reconciliation • Data maintenance • Data protection • Data provenance • Data surrogation • Data bloat • Lack of knowledge of original purpose • Lack of good data dictionaries • Ontological development for • intelligent searches

9. Key Data Issue: Barriers to Reuse • Lack of knowledge about existence of legacy data • Security restrictions • Quality of metadata (e.g., failure to document conditions of collection) • Varying definitions, language, measurement instruments • Form of accessible data • Rapid change of technical data • Fear (e.g., misuse, misunderstanding, adverse consequences)

10. Why Do We Care About Data? • Frequently drives solution strategy (e.g., study schedule, costs) • Determines surrogate vice ideal MoMs • Constrains viable scenarios • Drives treatment of key factors; e.g., • Treatment of human factors • Addressing organizational issues • Affects selection of tools

11. What the Team Needs to Know About Data • Data needs/data structure • Preferred • Necessary • Available • Data accessibility • Ownership • Security issues • Costs (buy, collect, generate)

12. Comments on Selected Data Challenges • Overall goals • Make data visible, accessible, and understandable across military organizations and beyond • Obtaining data • Existing data: this entails finding, organizing, verifying, processing, and converting data • Non-available data (e.g., new concepts of operations) • Tap SMEs, results of simulations,… • Document via metadata • Replace expert opinion with empirical data (e.g., experiments, operational data) ASAP • Data conversion • Initial: vague, uncertain, incomplete, contradictory, soft • Desired: sharp, certain, complete, consistent, hard • Be explicit about how you converted the data!

13. Selected Best Practices • Near-term • Employ emerging “analytical baselines” (DODD 8260) (subsuming scenarios, concept of operations, integrated data) • Find and reuse data to the extent feasible • Use metadata to document key data actions (e.g., implement DoD Discovery Metadata Standard (DDMS)) -- and update appropriately! • Longer-term • Establish a C2 Assessment Community of Interest • Align data processes and toolsets, as early as possible, with the C2 systems community (e.g., use same Information Resource Dictionary System) • Employ data engineering to gather, organize, convert, and VV&C available data

14. Summary • The data problem is complex and enormous … and increasing in both complexity and size! • The community has taken significant initial steps to address the problem; e.g., • Issued new directives, instructions (e.g., DoDD 8260) • Created new organizations (e.g., Joint Analytic Data Management Steering Committee (JADMSC)) • Formulated a framework based on the concept of enterprise, community of interest, and private data • Promulgated new tools, standards (e.g., DDMS) • However, in order to make further substantive improvements, we have to • Transform the culture (e.g., by implementing incentives, overcoming disincentives) • Educate & train the users, providers of data -- and the decisionmaker! • Implement new processes (e.g., work the metadata problem)