National Experimentation, Testing, Training, and Technology Conference

1. National Experimentation, Testing, • Training, and Technology Conference • A Perspective on • Life-Cycle Evaluations • Mr. David Duma • Principal Deputy Director • Operational Test and Evaluation • July 23, 2003

2. Secretary Rumsfeld on TransformationAugust 2002 • SECDEF’s Top Ten Priorities • Strengthen Joint Warfighting Capabilities • “. . . we are requiring that a joint operational concept be fashioned that will then be used to test proposals from the various Services to see to the extent to which they can be rationalized and harmonized to fit within a joint operation concept.”

3. Increased Emphasis OnJoint Warfighting Capabilities Joint Warfighting Joint Operational Concepts with Integrated Architectures Research USA Research USN Research USAF EXPERIMENT EXPERIMENT EXPERIMENT Coalition Forces EXPERIMENT Sec of the Navy Sec of the Air Force Combatant Commanders Sec of the Army Joint Forces Service Forces TEST TRAIN TEST TRAIN TEST TRAIN TEST TRAIN Train & Equip Joint Forces Train & Equip USA Train & Equip USN Train & Equip USAF

4. A Life-Cycle Evaluation ContinuumCJCSI/M 3170 / DoDD/I 5000 Overarching Policy Overarching Policy NSS/NMS/Joint vision NSS/NMS/Joint vision Joint Concept of Operations Joint Concept of Operations Functional (Program (Program Functional Area Functional Area Area Functional Concept Functional Concept Initiation) Initiation) Analysis FOC FOC B B C C A A IOC IOC Functional Integrated Architecture Integrated Architecture Area Analysis B B B C C C A Concept Concept Technology Technology System Development System Development Operations & Operations & Production & Deployment Production & Deployment D Refinement Refinement Development Development & Demonstration & Demonstration Support Support O - - Materiel Materiel Analysis of Analysis of Critical Critical FRP FRP T D Concept Concept Process Process Materiel Materiel Design Design Decision Decision LRIP/IOT&E LRIP M O Decision Decision Approaches Approaches Review Review Review Review L T ICD ICD ICD M P Pre Pre - - Systems Acquisition Systems Acquisition Systems Acquisition Systems Acquisition Sustainment Sustainment DOTLPF DOTLPF L F Process Process JROC JROC P F JROC JROC ACTDs/ATDs CT/DT/LF/OA IOT&E FOT&E TRAINING JT&E (I (Increment Initiation) Initiation) FOC FOC B B C C IOC IOC Technology Technology System Development System Development Operations & Operations & Production & Deployment Production & Deployment Development Development & Demonstration & Demonstration Support Support Critical Critical FRP FRP Design Design Decision Decision LRIP LRIP/IOT&E Review Review Review Review Pre-Systems Acquisition Pre-Systems Acquisition Systems Acquisition Sustainment (Increment (I Initiation) Initiation) B B C C IOC IOC Technology Technology System Development System Development Production & Deployment Production & Deployment Development Development & Demonstration & Demonstration Critical Critical FRP FRP Design Design Decision Decision LRIP LRIP/IOT&E Review Review Review Review Systems Acquisition ACTDs/ATDs CT/DT/LF/OA IOT&E FOT&E ConceptContractCombat STUDIES/EXPERIMENTS ACTDs/ATDs CT/DT/LF/OA IOT&E FOT&E TRAINING JT&E The Tools Are Different But The Principles Are the Same

5. Applying the Scientific Method for Life-Cycle Evaluations “. . . evaluations should draw upon principles founded in the scientific method and design of experiments. Items such as prediction (hypothesis), planning (test design), data collection (test event), analysis (data verification), and reporting (conclusion) should all contribute to providing the most credible, insightful, and valuable evaluation possible.“ Tom Christie June 2002

6. Essential Elements of an Evaluation Continuum Four things that will not change as a result of Training Transformation • The need to: • Provide operational realism • Use smart opposing forces • Compare to ground truth • Learn from feedback • Dr. David Chu • September 2002

7. An Evaluation Continuum that Provides Operational Realism • Evaluate the degree of mission success • Concepts—DOTMLPF—force structure • OT&E—DOTMLPF—units w/ new systems • JNTC—DOTMLPF—units w/ fielded systems • Link realism throughout the life-cycle • Consider assumptions, limitations, and safety • Do Missions with operational and tactical tasks • Use a challenging battlespace environment • Use clutter and counter measures • Use info operations and computer network attacks • Resource data gathering adequately • Use live, virtual, and constructive inputs • Use targets for the end game

8. An Evaluation Continuum that Uses Smart Opposing Forces • Evaluate w/ an enemy who votes—DOTMLPF • Is manned and organized realistically • Is trained and equipped realistically • Has a mission and means too • Do not unrealistically constrain the enemy • Rules of engagement permit full range of action • Encourage innovative and aggressive TTPs • Use deception and countermeasures • Play unit strengths against unit weaknesses

9. An Evaluation Continuum that Compares Results to Ground Truth • Evaluate against a standard • Measure current capabilities • Mission level measures of success • Instrumentation • Record current capabilities • Formal and accessible • Validated and updated • Measure results for comparison • Use consistent measures throughout life-cycle • Concepts, AoAs, ICDs, CDD, CPD • ACTDs/ATDs, CT/DT/OT, JT&E, JNTC • Use consistent instrumentation • Fixed, mobile, non-intrusive (embedded) • Life-cycle viewpoint

10. An Evaluation Continuum that Learns from Feedback • Evaluate with improvement in mind • Use what we now know to get better • Look to the future • Build formal feedback loops • Use results to refine perceptions • Refine warfighting concepts • Adjust capability expectations • Record results for future use and comparison • Validate & update demonstrated performance • Build an accessible performance database • Adjust measures and methods • Incorporate new technologies • Develop new instrumentation

11. An Evaluation Continuum that Draws from Strengths • So what does the OT&E community offer? • Mission perspective—Realism • Measures of success—Rigor • Instrumentation—Ground truth • Instrumentation • MRTFB—laboratory and field instrumentation • CTEIP—new test capabilities • T&E/S&T—new test technologies • LFT&T—test & training capabilities

12. An Evaluation Continuum that Gets at the Ground Truth • Examples of existing NET3 Technologies • MRTFB • Threats (FMA & surrogates), TSPI, GPS, telemetry • Real time casualty assessment (ground-ground) • CTEIP • Foundation Initiative 2010—TENA (BIC) • Advanced Range Telemetry—Half the RF Spectrum • T&E/S&T • RF spectrum efficiency • Hypersonic measurement • LFT&T • Virtual Target and Range (VITAR) • Project Grace—MIG 29, AA10, AA11

13. An Evaluation Continuum that Gets at the Ground Truth • Examples of needed NET3 technologies • Non-intrusive instrumentation • Micro Electrical Mechanical Systems (MEMS) • Hardened Subminiature Telemetry & Sensor System • Real time casualty assessment (air-ground) • Directed Energy methods and measures • Hypersonic methods and measures • Comprehensive performance database • An invitation to trainers and experimenters • Help foster a DoD life-cycle evaluation continuum • Help guide instrumentation development • Help develop new methods and measures

14. A Life-Cycle Evaluation ContinuumNeeds a Common Language Overarching Policy Overarching Policy NSS/NMS/Joint vision NSS/NMS/Joint vision Joint Concept of Operations Joint Concept of Operations Functional (Program (Program Functional Area Functional Area Area Functional Concept Functional Concept Initiation) Initiation) Analysis FOC FOC B B C C A A IOC IOC Functional Integrated Architecture Integrated Architecture Area Analysis B B B C C C A Concept Concept Technology Technology System Development System Development Operations & Operations & Production & Deployment Production & Deployment D Refinement Refinement Development Development & Demonstration & Demonstration Support Support O - - Materiel Materiel Analysis of Analysis of Critical Critical FRP FRP T D Concept Concept Process Process Materiel Materiel Design Design Decision Decision LRIP/IOT&E LRIP M O Decision Decision Approaches Approaches Review Review Review Review L T ICD ICD ICD M P Pre Pre - - Systems Acquisition Systems Acquisition Systems Acquisition Systems Acquisition Sustainment Sustainment DOTLPF DOTLPF L F Process Process JROC JROC P F JROC JROC ACTDs/ATDs CT/DT/LF/OA IOT&E FOT&E TRAINING JT&E (I (Increment Initiation) Initiation) FOC FOC B B C C IOC IOC Technology Technology System Development System Development Operations & Operations & Production & Deployment Production & Deployment Development Development & Demonstration & Demonstration Support Support Critical Critical FRP FRP Design Design Decision Decision LRIP LRIP/IOT&E Review Review Review Review Pre-Systems Acquisition Pre-Systems Acquisition Systems Acquisition Sustainment (Increment (I Initiation) Initiation) B B C C IOC IOC Technology Technology System Development System Development Production & Deployment Production & Deployment Development Development & Demonstration & Demonstration Critical Critical FRP FRP Design Design Decision Decision LRIP LRIP/IOT&E Review Review Review Review Systems Acquisition ACTDs/ATDs CT/DT/LF/OA IOT&E FOT&E ConceptContractCombat STUDIES/EXPERIMENTS ACTDs/ATDs CT/DT/LF/OA IOT&E FOT&E TRAINING JT&E • It’s All About the Warfighter • Train as we fight • Test as we fight • Refine how we fight • Life-Cycle Communication • Understand you • Understand me • Mission Essential Tasks • UJTL/JMETLs/SMETLs • Translate to Contract Specs

15. National Experimentation, Testing, • Training, and Technology Conference • A Perspective on • Life-Cycle Evaluations • Questions?