CHALLENGES WITH QUANTIFYING THE QUALITATIVE

1. CHALLENGES WITH QUANTIFYING THE QUALITATIVE Presented to:ONR Workshop on Human Interactions in Irregular Warfare as a Complex System Atlanta, GA 13-14 April 2011 In collaboration with: Elizabeth Whitaker, Erica Briscoe, Ethan Trewhitt, Georgia Tech Kevin Murphy, Frank Ritter, John Horgan, Penn State Caroline Kennedy-Pipe, Univ. of Hull Presented by:Dr. Lora Weiss Georgia Tech Research Institute Lora.Weiss@gtri.gatech.edu

2. LINKING US-UK EXPERTISE(for understanding IED perpetration in Iraq) • Objective • Provide a methodology to scientifically capture, evaluate, and predict large-scale behaviors of potential IED developers before they have successfully deployed devices • Elicit information from UK subject matter experts, who have had different experiences on their homeland • Develop analytic tools to conduct quantitative and qualitative analysis of potential interdiction points SMEs SMEs Evaluation Influence Models System Dynamics Models Agent-based Models Doctrine Models Knowledge Engineering Literature Modeling Scenarios • Model Considerations • Incomplete Data • Data Provenance • Data Uncertainty • Data Perishability

3. Knowledge Engineering Instrument

4. Example Interview Results Individual Common Motivations Activities Common Goals Results “Western” Individual Individual Religion Individual Goal Activities Results Money “Non-Western” Individual Goal Power Individual Goal Common End-state vs. Individual Motivations • Management and planning within IED “teams” are different than in Western civilization • Participants are not necessarily focused on an end-state. Instead individual motivations (that may differ) are manipulated toward the individual’s end goals.

5. Interview Results - 2 IRA Iraqi Insurgency Activities change by being monitored vs. concentrating on technical execution Game-like planning Little attempt to Influence monitoring • Crucial differences: • IRA was aware they were being watched and operated in a manner to “fool” their pursuers • Iraqi insurgency less of a “game-like” attitude and is more concentrated on purely technical aspects Counter-IED Monitoring Counter-IED Monitoring

6. Interview Results - 3 Current Behavior Normal Behavior Comparison to spot differences Missing Normal Behavior Useful information often lost because no explicit sharing of stories when units transfer Unusual Anomalous Behavior Record and Share Stories • This information is usually subtle and not directly transcribable, e.g., noticing what is not normal about an environment (social or physical) • Military personnel notice things that are different and have a hard time putting their fingers on exactly what that is

7. Interview Results - 4 Personnel Recruiting Religious Motivation Monetary Motivation Power Motivation Motivation varies among lower level participants • For lower level participants (beneath management), motivation is most often monetary or peer involvement • Experts are conflicted as to whether religion is actually a motivator or just used as ‘clean’ explanation

8. Mind Map for Preliminary Knowledge Structuring

9. From Knowledge Engineering to Modeling System Dynamics • Methodology for evaluating of complex systems over time • Represent causal relationships and feedback • Stocks and flows represent the movement of items, materials, people, or abstract concepts • Easy experimentation with changes in structure, inputs, conditions

10. Simplified Core Model Central features related to IED perpetration in Iraq

11. Materials and Supplies • Stock – Materials & Supplies - represents the inventory of generalized materials and supplies of insurgent groups in the area • Input Flow - Gathering - represents actions that cause the accumulation of materials and supplies • Output Flow - Consumption - represents the use of these materials and supplies in the construction of IEDs

12. IED Process • Four stages of IED Process: Constructed, Inventory, Emplaced, Detonated • Flows between stocks represent transitions from one stage to another • The Disrupted IED stock and its related flows, Early, Middle and Late Disruption represent the destruction of IEDs by counter-IED efforts.

13. Personnel • Represent the transition of a sympathizer into active participation within a terrorist group • Radicalization represents transition of a person from within the general population to the Grey Population. • A previously neutral person taking a position of sympathy for insurgent beliefs • Deradicalization is the reverse of this, when a person loses sympathy for the insurgency • As a person becomes an active participant in the IED process, this is represented as Recruitment • Death and Disengagement indicate that an active insurgent has left the group in one way or another.

14. Integrated Model

15. Incorporation of Submodels

16. Personnel

17. Two Radicalization Submodels Based on method of Bartolomei, J., Casebeer, W., & Thomas, T. (2004) Derived from SME input

18. Representing Culture Influences • Complex socio-cultural computational models include both quantitative and qualitative data • Qualitative • Interviews with perpetrators • Opinions of SMEs • Broad social, psychological theories • Quantitative • Demographics • Economic Factors • Surveys Want to start understanding the interactions of all these influences  What-If Analyses

19. What-if Analysis Enable analysts to • Experiment with different sets of parameters, variables, and relationships • Explore results of • Events within our control (military actions, policies, diplomatic decisions) • Events not within our control (weather, crop production, actions of others) Potential Events Impact of Change Models Environmental Context Evaluate Scenarios

20. Modeling Approaches Across The Sciences Figure from: G. Zacharias, J. MacMillan, and S. Van Hemel (eds), Behavioral Modeling and Simulation, National Research Council, 2008.

21. Qualitative Socio-Cultural Data • External Influences on Behaviors • Political Attitudes, • Influences • Policies • Opinions, Experiences of SMEs • Interviews with Individuals or Groups Being Modeled • Cultural Descriptions • Social Theories • Observed Behavior • Psychological Theories

22. Quantitative Socio-Cultural Data Economic Factors • Environmental Measurements Survey Data Demographics Polls • Psychometric Measurements Census Data

23. Representing Qualitative Data in a Computational Model • Accepted measurement scales may not exist • Modelers may need to create fuzzy or landmark values for abstract concepts

24. What Kind of Data Does Your Model Need? • What types of socio-cultural data does your model need? • What would you do if you never got it? • What are realistic substitutes and workarounds?

25. Realistic Substitutes and Workarounds

26. Dealing with Uncertainty in Socio-Cultural Data • Where does uncertainty occur? • Uncertainty in descriptions of attitudes, cultures, behaviors • Uncertainty in measurements (physical measurements or survey instruments) • Uncertainty in descriptions of historical situations • Uncertainty inherent in human behavior • Variations in human choices given the same culture and situation • What approaches exist for dealing with uncertainty? • Probabilistic approaches, random variables • Representations of likelihood (other than strict probability) • Techniques for combining certainty values

27. Data Provenance • Provenance: Where did the data come from, who collected it, how was it collected, under what circumstances, and what was the context? • The model user should understand the provenance of data in order to determine how appropriate it is for a particular use. • Data from an authoritative source is not automatically more useful than data from an unreliable source. • Data known with a high degree of certainty may not be the data that leads to recognition of unexpected behaviors. • Once a piece of information has been confirmed and ‘hits the news’, it may no longer provide information that can be acted upon. • In contrast, rumors about conspiracies, although potentially false, are sources of information that may allow intervention to prevent catastrophic events.

28. Data Perishability and Missing Data • Perishability: How long will this data be valid? • The importance of knowing when to remove data from a model and recognizing that behaviors change and adapt • Model representations in this space need to be those that can be made robust against missing features.

29. Summary Adopt Best Practices for Integrating Qualitative Data into Quantitative Models