Evaluation of Training Effectiveness: Tactical Decision Making Systems and Training of Instructors

1. Evaluation of Training Effectiveness: Tactical Decision Making Systems and Training of Instructors Amela Sadagic, PhD

2. Agenda • Objective • Overarching Approaches • Hypothesis • Train the Trainers • Pilot Study • Final TEE Study • Additional Contributions • Q & A

3. Research Question & Answer • Issues of availability of good training simulations cannot be treated separately from the issues of availability of equally good training methodologies and pedagogies. • Q: When we work with the training systems, should we even consider doing studies that do not employ well designed training methodologies and pedagogies? • A: We believe we should not.

4. Objectives • Primary: TEE study of simulations used to train tactical decision-making combat skills. • Preliminary study of combining different training systems and tools. • Develop the guidelines for optimal training approaches and pedagogies that most effectively serve selected training objectives and audience. • Identify training opportunities and potential for extended use of TDS-s.

5. Glossary • TDG: Tactical Decision Games pen-and-paper technique • TDS: Tactical Decision-making Simulation computer-based training simulations that use gaming technologies • CCM: Close Combat Marine Multiuser TDS, navigation through 2D world with top-down view of the terrain • IOC: Infantry Officers Course, USMC • TBS: The Basic School, USMC

6. Overarching Approaches • Directly connect our research with: • Institution that defines the training requirements and makes system recommendations: TECOM (USMC), • Institutions that provide the training: The Basic School - TBS and Infantry Officers Course - IOC (both USMC). • Support USMC strategy of Distributed Operations, • Address specific training needs of the next generation of instructors (‘train the trainers’), • Explore novel training approaches and pedagogies, • Allow for a synergetic mix of old systems and methods (TDG) and new systems and methods (TDS).

7. Hypothesis • Primary: The strategic use of virtual training simulations does contribute towards higher effectiveness in training, affecting learning results and motivation. • ‘Early majority’ and ‘late majority’ groups of users will need more than just having the physical access to training systems and tools.

8. Train the Trainers • Acquire personal skills and knowledge but also know how to teach / instruct other people: • gradually increase the complexity (crawl-walk-run): fight alone -> ‘buddy-on-buddy’ -> ‘force-on-force’ • role changing: be both team leader and a team member, • evaluate someone else’s performance, • multiple AAR opportunities, • add a motivation ingredient. • Issue: a ‘gaming’ behavior • solution: instead of fighting the ‘destructive’ tendencies (use of non-doctrinal techniques), redirect them to constructive direction

9. Pilot Study: 12 candidates from IOC

10. Pilot Study: 12 candidates from IOC • av. 24 years old, all 2nd Lt., av. 29.5 months in service • Prior use of training systems: 8 YES (5.3 hours total) • Game experience: 10 YES (most used several times a year), first person shooters (9), racing (5), other sports (5) • Own personal computer: everyone! (av. 5.3 years) • What do they use it for: email + web browsing • Other applications: Microsoft Office Suite (4) but rarely used • Skills they have most difficulties with: • Direct a CAS (12!), Employ support arms (4), Direct the employment of 60mm/81mm mortars for offensive ops.(4) and defensive ops (3)

11. Pilot Study: 12 candidates from IOC • Post-session results: • Majority agreed that 2 hours of training was not enough, • Majority reported no change in skill level for most of the skills, • For some skills they reported higher initial proficiency level than the level they reported at the end… yet they thought they improved those skills with training they just received: • initial skill level overstated • Could we use TDS as a validation tool (‘How well are we doing with our training?’)

12. Final TEE Study (IOC): a plan • Select training objectives that can be effectively conquered in a given time-frame, • Integrate selected training approaches and pedagogies, • One group of IOC candidates to use TDGs and another group TDS. Then they switch. • Exposure: 2 training sessions, 3 hours each. • Conduct evaluation of learned skills after each session. • Collect both subjective and objective performance measurements.

13. Additional Contributions • Assist IOC in optimizing current training program with TDS, • Assist TBS in defining new POI (a part associated with the use of TDSs), • Advise both institutions on how to record and follow up the performances exhibited by all candidates that will go through the schools. • identify performance trends and if needed address them with appropriate training interventions.

14. To sum up • If one wants to serve the needs of majority of users: • Do not just ‘thick the box’ - dedicate a solid amount of time to this type of training, and offer it to the people that need it. • Explore a synergy of the old (proven) and new (proven & perhaps exciting) training techniques and systems. • Insert motivation elements. • Fully integrate technologies/systems with best training approaches and pedagogies. • Provide a ‘full package’.

15. Special thanks to: Sponsor: Navy Modeling & Simulation Management Office (NAVMSMO) Partners in the project: USMC TECOM, IOC and TBS, Quantico, VA Maj McDonough

16. Q & A Contact: asadagic@nps.edu / ph: 831.656.3819 www.movesinstitute.org/~amela