Conceptual Model: Assumption Testing

1. Conceptual Model:Assumption Testing Robert Eberth Sanderling Research Corporation 25 March 2008 robert.eberth.src@cox.net 571.257.6997

2. Conceptual Model: Assumption Testing • Precepts: • Overarching purpose is to test the ABS Val framework • ABS Val process based on modern scientific method and its falsifiability criterion • The null hypothesis is the research hypothesis that the model is valid (“sufficiently accurate”) for its specific intended application • Objective then is to falsify the null hypothesis • Failure to falsify the null does not prove the model valid for the specific intended purpose, but should increase confidence in its validity • Degree of confidence then depends on the rigor and power of the tests applied

3. Conceptual Model: Assumption Testing (cont’d) • Plan: • I.D. the analytic questions at hand, their metrics, and degree that results are expected to shape decisions • Detailed review of all documentation • Interview Application Sponsor • With the App Sponsor, I.D. the referent; i.e., the proxy for the real world for accuracy comparisons • With the App Sponsor, I.D. the accreditation criteria • How “accurate” must the model be • How can/will accuracy be determined (quant/qual) • Criteria must establish lower bounds of acceptability

4. Conceptual Model: Assumption Testing (cont’d) • Assess the validity of the referent • Confirm that no alternative – and preferable – referent is available or could be made available • Assumption test the referent (for other than empirical datasets): • I.D./derive inherent assumptions • Perform logical verification – adequacy and correctness vis-à-vis underlying theory and assumptions • I.D. the operational implications of the assumptions • Determine the bounds of validity imposed on the app’s problem space and on the model’s validity assessment by the referent’s assumptions • Determine whether the operational implications and bounds of validity are acceptable to the Application Sponsor • Independent SME reviews (ideally, contrarian reviews)

5. Conceptual Model: Assumption Testing (cont’d) • Assess the validity of the conceptual model • Potentially three separate assessments • Theoretic sub-model • Mathematic sub-model (if it exists) • Algorithmic sub-model • Each in turn receives same assessment techniques • Logical verification – determining sub-model is an adequate and correct implementation of its predecessor. • Assumption testing • I.D./derive the assumptions that are inherent to/embedded in the sub-model • Determine the operational implications of the identified assumptions in the context of the particular application

6. Conceptual Model: Assumption Testing (cont’d) • Determine the bounds of validity of the model that are the result of the identified assumptions • Determine whether the operational implications and bounds of validity are acceptable to the Application Sponsor for the intended application • For some models, it may prove necessary to reverse-engineer one or more sub-models from later models. It may even be necessary to reverse-engineer the conceptual model, or portions of it, from source code. • The referent serves as the predecessor for the theoretic sub-model

7. Results • BLUF: Framework “worked,” (or more correctly, “is working”), but: • More work needed on how to select and assess the validity of the referent when empirical data are not available for use as the referent • Framework needs templates • Groundwork: • This portion of effort dealt with the theoretic model. Algorithmic model work yet to come • Began with study of Pythagoras Manual and related detailed discussions with Edd Bitinas. However, Pythagoras itself was not assessed (V&V assumed) • Interviews w/ LT Robin Sparling, USN, the COIN study’s Project Officer (in place of Steve Stephens), and study of several study-related documents she provided

8. Results (cont’d) • Interview w/ Dr. Akst, the Application Sponsor • Purpose was to “make headway in developing a COIN model.” • Did not specify an ABS, let alone Pythagoras • Approved recommendation of using “sea versus land basing” as study’s analytic question, but did not specify it going in. • Approved stated Marine missions, and O.K. with implied mission • Insisted study must use real-world dataset

9. Results (cont’d) • Findings: • Multiple objectives: • OAD was to “make headway” in developing a COIN model • NGMS was tasked to determine whether and how Pythagoras could be used to support IW analyses • Study at hand had the analytic objective of determining whether it was best to leave the MAGTF ashore or afloat in a Columbian HA/DR/Security scenario • USMC missions in Columbian scenario: • Refugee camp security • Humanitarian Assistance • Disaster Relief

10. Results (cont’d) • But, study team found no way to directly evaluate the effectiveness of mission performance • Thus decided to use allegiance changes of population segments among several distinct affiliation possibilities – thus producing an “implied mission” of keeping the bad guys from gaining strength • Stated as “Do not allow illicit organizations to take advantage of situation” • But may also imply that ABSs in general and Pythagoras in particular cannot support traditional MOEs of mission performance • Alternative possibility: When the population is the Center of Gravity, may affiliation changes be the best MOE for mission performance (and not just for HA/DR)? • NB: The population is NOT always the COG of an insurgency – it depends on the goal of the insurgency

11. Results (cont’d) • Several assumptions then had a large impact: • Modeling the transitions among affiliations as a Markov process (a “memoryless” process) • Constant transition probabilities across all time steps (except those during the Marines time in-country) • Constant transition probabilities across all time steps while the Marines were in-country (although different probabilities from the baseline) • Initial indications are that the above assumptions absolutely pre-determined the results and in a predictable way (i.e., the model became deterministic if allowed to run to steady-state)

12. Results (cont’d) • Unfortunately, that may mean that OAD can’t make a solid determination on the usefulness of Pythagoras in the IW or COIN context from this particular application • It also may mean that the answer to the one analytic question (afloat or ashore) depends entirely on the methodology used to develop the transition probabilities – the “influence estimation” and “salience” parameters • And those are suspect because of potential bias in data collection/analysis methodology (semantic differential) and distinction between “data” and “context”

13. Results (cont’d) • Initial indications wrt the study (again, only from assessment of the theoretical model, so subject to change): • Probably cannot yet give a defensible answer to the afloat/ashore analytic question • Implementation assumptions too limiting • Semantic differential data collection/analysis methodology far too suspect • BUT, study may represent a huge leap forward in IW analysis • Could/should cause a re-evaluation of COGs and MOEs for IW environments • Could/should lead to a series of studies on semantic differential and alternative methodologies for capturing the propensity of persons to change affiliations, particularly in response to actions/events rather than just presence