Multinational Experiment 4 on Effects Based Approach to Operations A CFEC Report

1. Multinational Experiment 4 onEffects Based Approach to OperationsA CFEC Report Philip S. E. Farrell, Ph.D. A presentation for IAMWG Meeting Friday 11 May 2007

2. Presentation • Introduction – Human Experimentation • MNE 4 CFEC Report • Discussion – Can war-fighting experimentation be done in a controlled laboratory setting?

3. Introduction Not “The Bear” Now “The Ball of Snakes” New concepts need to be tested

4. Scientific Method High School Science Report • Background • Purpose • Hypothesis (related to concept) • Analysis • Apparatus • Procedures • Results • Discussion • Conclusions

5. Human Experimentation

6. Human Experimentation Job Similarity Index concept Crew Selection Experiment (2006) TTPs Recommendations

7. Human Experimentation in MNE 4 • Too many people? • Too many variables? • Too many confounds? • Too many constraints?

8. MNE 4 Experiment Effects Based Approaches to Operations Refinement and Assessment of EBAO POT actions disturbances Planning (desired effects) Execution Desired End State + desired actions + + _ Operational Environment World states Assessment (effects) Current situations

9. MNE 4 Experiment (CONOPS 0.9) Planning RG Execution Ra Rs Re A D + + + + + ESA ED SPR PREPO C&C ADRM _ Ca _ _ Cs + IDI “action” comparators CA “situation” and “effects” comparators Ce + EBAP W Analysis Function KBD, SOSA, KM S MMA AAMOP AAMOE QCE Assessment

10. MNE 4 Experiment Experiment Aim Experiment Objectives Hypothesis Critical Operational Issues Measures Data Experiment Execution Data Collection

11. MNE 4 Experiment Competing Aims! • to explore concepts and supporting tools for EBAO • to address the Canadian MNE 4 objectives

12. MNE 4 Experiment • Canadian Objectives • Refine, assess, and analyze elements of EBAO that Canada and the CF could apply to achieve integrated strategic effort in operations. • Determine whether EBAO represents a means of applying the Canadian 3-D approach to operations • Develop and refine a Multinational Knowledge Management Plan that can be tailored and applied to meet national operational requirements

13. MNE 4 Experiment The Hypothesis of this experiment was: If an operational commander and staff employ effects based operations, then they will have the ability to: Create a holistic understanding (KBD) Develop and refine an operational plan (EBP) Harmonize the full spectrum of DIME planning (MNIG) Coordinate, direct, and monitor operations (EBE) Assess actions, effects, and end state (EBA)

14. MNE 4 Experiment

15. MNE 4 Experiment • Experimental Design

16. MNE 4 Experiment Types of Analyses • Observations • JOT (Joint Observation Tool) • JDCAT (Joint Battle Centre Data Collection Analysis Tool)

17. MNE 4 Experiment Types of Analyses • User surveys and network data

18. MNE 4 Experiment Types of Analyses • Box Plots (KBD) Figure C.1 Distribution of correct responses for SA surveys, by organization

19. MNE 4 Experiment Types of Analyses • Multi-Dimensional Scaling (KBD) Figure C.4: Multi-dimensional scaling of KBD3 answers by Group Reference Point (KBD 3) Reported same incorrect answers

20. MNE 4 Experiment Types of Analyses • Count data (KM)

21. Q3.2b. To what degree is the decision of the leader contingent on the work and advice of others? (this question was for those who did not answer Leaderless Team in Q3.2a) Mann-Whitney U = 2212, p < .07 Not at all Little Somewhat Much Completely Not applicable 2 NRF 0% 5% 44% 36% 11% 4% 2 = 38.18, p < .001 CTF 1% 8% 19% 54% 15% 3% 2 = 110.13, p < .001 MNE 4 Experiment Types of Analyses • Whitney – U (Culture)

22. Date Name Vector 28 Feb v1 (7, 5, 6, 7) 2 Mar v2 (7, 6, 5, 7) 7 Mar v3 (7, 6, 7, 7) 9 Mar v4 (7, 7, 7, 6) 14 Mar v5 (7, 6, 7, 7) 16 Mar v6 (7, 7, 7, 6) Reference r (7, 7, 7, 7) MNE 4 Experiment Types of Analyses • Vector Method (Common Intent)

23. v1 v2 v3 v4 v5 v6 r v1 0 6.4 4.7 10.0 4.7 10.0 7.6 v2 6.4 0 7.8 10.0 7.8 10.0 7.6 v3 4.7 7.8 0 6.0 0.0 6.0 3.7 v4 10.0 10.0 6.0 0 6.0 0.0 3.7 v5 4.7 7.8 0.0 6.0 0 6.0 3.7 v6 10.0 10.0 6.0 0.0 6.0 0 3.7 r 7.6 7.6 3.7 3.7 3.7 3.7 0 MNE 4 Experiment Types of Analyses • Vector Method (Common Intent) Table 3. Angle between vector pairs in degrees (Duncan & Farrell, 2007)

24. MNE 4 Experiment • Results, Discussion, Conclusions, and Recommendations are just as fuzzy as the objectives and hypothesis.

25. MNE 4 Experiment Insights related to Canadian Objectives • The notion of planning, execution, and assessment in an iterative fashion until the Current State reaches the End State is a robust concept. • The relationship between MNIG, EBAO, and the 3-D concepts requires further development and experimentation. • Some of the basic KM and KBD definitions and concepts still have not been resolved.

26. MNE 4 Experiment Recommendations • More research is required!

27. Discussion Can war-fighting experimentation be done in a laboratory controlled setting?

28. Discussion Military buy-in vs. Scientific reputation • Do you need SMEs to conduct the experiment?

29. Discussion Human Experimentation Ethics • Is voluntary consent necessary?

30. Discussion political Considerations • Who makes the decisions?

31. Discussion Concept complexity • Are there too many variables?

32. Discussion Experimental Design • Whatever happened to Hypothesis-testing?

33. Summary • Human Experimentation is not new. • MNE 4 has refined our processes for war-fighting experimentation. • We need to pay attention to the scientific method if our recommendations are to be credible

34. Questions