DOT&E Scorecard: Wins & Losses, Post-game Report

1. DOT&E Scorecard: Wins & Losses, Post-game Report Ernest.Seglie@osd.mil

2. Scorecard • Wins? • Losses? • Post-Game? • Whose Scorecard? • We give scores • We get scored

3. 3 Who What 4 I don’t know 5 Why 7

4. The scores we give The scores we give

5. DOT&E scores major acquisition programs. Reports to Congress on IOT&E: 110. Score is whether the test was adequate and whether the test confirmed that the system actually tested is effective and suitable for combat.

6. 3-Dec-84 TOMAHAWK (TASM & TLAM-N) yes yes yes Mr. Weinberger 27-Dec-84 M1A1 Abrams Tank yes yes yes Mr. Weinberger 4-Mar-85 Standard Missile Block II yes yes yes BG Hall 24-Aug-85 Sergeant York (DIVAD) yes no no Mr. Krings 30-Sep-85 OH-58D Army Helicopter Improvement Program yes yes (-) yes (-) Mr. Krings 7-Oct-85 AV-8B Harrier yes yes yes Mr. Krings 4-Dec-85 TOMAHAWK (TLAM-C) yes yes yes Mr. Krings 12-Mar-86 E-6A TACAMO Operational Assessment yes potential potential Mr. Krings 19-Mar-86 AEGIS/CG-47 Class Cruiser yes yes yes Mr. Krings 16-Apr-86 AGM-65D Infrared Maverick FOT&E yes yes yes Mr. Krings Arleigh Burke (DDG51) OA 17-Sep-86 yes yes yes Mr. Krings 30-Oct-86 Tactical Army Combat Service Support Computer System yes yes yes Mr. Krings 24-Nov-86 Low Altitude Navigation & Targeting Infrared System no yes yes Mr. Krings 12-Jun-87 Landing Craft Air Cushion yes yes yes Mr. Krings ELF Commo System 25-Jun-87 yes yes yes Mr. Krings 24-Aug-87 S-3A Weapon System Improvement Program (S-3B) no potential no Mr. Krings 6-Oct-87 TRIDENT II (D-5) Strategic Weapons System OA yes low risk low risk Mr. Krings 1-Dec-87 Bradley (follow-up OA) yes potential potential Mr. Krings 1-Apr-88 CV Inner-Zone ASW Helo SH-60F yes potential potential Mr. Krings 11-May-88 MH-53E Helicopter yes marginal no Mr. Krings 13-Jun-88 S-3A Weapon System Improvement Program (S-3B) yes yes yes(-) Mr. Krings 30-Aug-88 Bigeye Interim Report no potential potential Mr. Krings 9-Sep-88 OH-58D AHIP Scout/Recon yes yes not tested Mr. Krings 15-Nov-88 Secure Telephone Unit (STU III) yes marginal marginal Mr. Krings 5-Dec-88 Mobile Subscriber Equipment FOT&E yes yes yes Mr. Krings 8-Dec-88 MK48 Advanced Capability Torpedo yes no yes Mr. Krings 14-Dec-88 M9 ACE yes yes marginal Mr. Krings 19-Dec-88 Low Altitude Navigation & Targeting Infrared System yes yes yes Mr. Krings 1-Jan-89 MK-48 ADCAP Torpedo yes yes yes yes 1-Feb-89 Phoenix (AIM-54C) yes yes(-) yes 25-Apr-89 AN/SQS-53C Sonar System Mr. Krings 24-May-89 yes yes yes Mr. Krings M939A2 5-Ton Truck 1-Jul-89 yes potential + potential - Mr. Adolph ASPJ OA LRIP Memo for USD(A) 1-Sep-89 JTIDS yes yes no Mr. Adolph 26-Mar-90 Pedestal Mounted Stinger yes yes yes(-) Mr. Duncan 1-Oct-90 Rolling Airframe Missile yes yes (-) no Mr. Duncan 1-Oct-90 LFT&E of Rolling Airframe Missile (RAM) yes yes 1-Oct-90 FAADS/LOS-F-H yes yes (-) no 1-Oct-90 LFT&E of FAADS/LOS-F-H 31-Oct-90 ATACMS yes yes(-) yes Mr. Duncan 1-Dec-90 LFT&E of ATACMS BLK I yes(-) 13-Dec-90 Single Channel Ground & Airborne Radio Channel yes yes yes Mr. Duncan 24-Jan-91 Advanced Cruise Missile yes yes yes(-) Mr. Duncan 29-Jan-91 TOMAHAWK TLAM/D yes yes yes Mr. Duncan 5-Feb-91 HH-65 Engine Study yes potential potential Mr. Duncan 26-Apr-91 AN/ALQ-184 & AN/ALM-233 yes yes yes Mr. Duncan 1-Jan-92 LFT&E of SM-2 BLK IIIA yes yes 21-Jan-92 NAVSTAR GPS User Equipment yes yes yes Mr. Duncan 6-Mar-92 B-1B Electronic Countermeasures Review yes no no Mr. Duncan 1-Apr-92 LFT&E of Torpedo MK-50 yes yes 1-Apr-92 LFT&E of HARM Improved Warhead yes yes 7-Apr-92 AIM-120A Advanced Medium Air-to-Air Missile yes yes(-) yes(-) Mr. Duncan 1-May-92 LFT&E of AIM-120A Advanced Medium Air-to-Air Missile yes yes(-) 1-Jun-92 LFT&E of TOMAHAWK TLAM/C yes(-) yes(-) 1-Jun-92 TOMAHAWK TLAM/C yes yes (-) yes (-) 2-Oct-92 Composite Health Care System yes potential potential Mr. Duncan 4-Dec-92 Airborne Self Protection Jammer OT-IID Memo for SecDef yes no no Mr. Duncan 15-Dec-92 ALR-56M Radar Warning Receiver yes yes yes Mr. Duncan C-17 EOA potential potential Mr. Frame Verlical Launch Antisubmarine Rocket FOT&E Mr. Frame 16-Dec-92 yes Milstar Navy EHF Satellite Program AN/USC-38(V) 2-Mar-93 yes no yes LFT&E of M830A1 120mm HEAT-MP-T Cartridge 29-Mar-93 yes yes yes Mr. Frame LFT&E of M109A6 Paladin Howitzer 1-Apr-93 yes yes 1-Apr-93 yes yes 9-Apr-93 M109A6 Paladin Howitzer yes yes yes Mr. Frame 30-Apr-93 Palletized Loading System yes marginal no Mr. Frame 1-May-93 LFT&E of Hellfire II yes yes(-) 1-May-93 LFT&E of M113A3 Spall Liner yes yes 1-Jul-93 DRAGON II LFT&E yes yes 19-Jul-93 Single Channel Ground & Airborne Radio Channel yes yes yes Mr. Frame 1-Feb-94 LFT&E of M919 25mm APFSDS-T yes yes 1-Feb-94 LFT&E of M829A2 APFSDS-T yes yes 3-Feb-94 Rolling Airframe Missile yes yes (-) no Mr. Frame 1-May-94 LFT&E of M1A2 Abrams Tank yes yes(-) 13-May-94 E-2C UDP yes yes yes Mr. Frame 9-Aug-94 M1A2 Abrams Tank yes yes no Mr. Frame 1-Nov-94 LFT&E of Sensor Fuzed Weapon (SFW) yes yes 12-Jan-95 T-45 Jet Flight Tng System yes yes yes Mr. Coyle 24-Feb-95 JTIDS Class 2/2H yes yes yes Mr. Coyle 23-Mar-95 B-1B Operational Readiness Test yes yes yes Mr. Coyle 8-Aug-95 Family of Medium Tacticval Vehicles yes yes yes Mr. Coyle 27-Sep-95 LFT&E of M903 Saboted Light Armor Penetrator yes(-) yes Mr. Coyle 12-Oct-95 AH-64D Longbow Apache/Longbow Hellfire yes yes yes Apache (-) Apache Hellfire Mr. Coyle 25-Oct-95 FAADC3I & Ground Based Sensor yes marginal yes Mr. Coyle 1-Nov-95 C-17 Globemaster III Airlift Aircraft yes yes yes yes(-) yes Mr. Coyle 31-Jan-96 AFATDS yes yes yes Mr. Coyle 11-Mar-96 MK 48 MODS ADCAP yes yes yes Mr. Coyle 10-May-96 Sensor Fuzed Weapon yes yes yes Mr. Coyle 30-May-96 LFT&E of M430A1 40-mm Cartridge yes(-) yes Mr. Coyle 1-Sep-96 ASPJ Limited Scope for Bosnia yes (-) yes (-) marginal Mr. Coyle 19-Sep-96 JSTARS yes yes (-) no Mr. Coyle 20-Sep-96 Standard Missile-2 Block IIIB yes yes yes Mr. Coyle 1-Apr-97 ATACMS IA Oper Assessment yes no no Mr. Coyle 13-May-97 Javelin yes yes yes yes yes Mr. Coyle 30-May-97 JTIDS - 2M yes yes no Mr. Coyle 8-Aug-97 Enhanced Position Location Reporting System yes yes (-) yes (-) Mr. Coyle 25-Sep-97 AWACS E-3 Radar System Improvement Program yes yes yes (-) Mr. Coyle 30-Jan-98 Ship Self Defense System yes yes (-) yes Mr. Coyle 1-Apr-98 ATACMS IA yes yes (-) yes yes yes(-) Mr. Coyle 1-Apr-98 ALR-67 (V)3 LRIP Certification yes potential potential Mr. Coyle 23-Apr-98 LFT&E of Armor Piercing Cartridges (M995 and M993) yes yes Mr. Coyle 7-Oct-98 JSTARS CGS Assessment yes no no Mr. Coyle 27-Oct-98 Joint Standoff Weapon - Baseline (JSOW) yes yes yes yes yes Mr. Coyle 20-Nov-98 Combat Service Support Control System (CSSCS) yes (-) no yes (-) Mr. Coyle 8-Dec-98 Close Combat Tactical Trainer yes yes no Mr. Coyle 13-Jan-99 B-1B Block D Conventional Munition Upgrade yes yes yes Mr. Coyle 22-Jan-99 SMART-T yes open issue no Mr. Coyle 1-Feb-99 LFT&E of B-1B Conventional Mission Upgrade Program yes(-) yes(-) Mr. Coyle 5-Mar-99 Cheyenne Mountain Upgrade yes yes yes Mr. Coyle 30-Jun-99 JSTARS CGS Assessment yes no no Mr. Coyle 16-Jul-99 LFT&E of XM-93 Wide Area Munition yes yes(-) Mr. Coyle 9-Aug-99 ALR-67(V3) Advanced Special Receiver yes yes yes Mr. Coyle 18-Oct-99 F-15 Fighter Data Link yes yes yes Mr. Coyle yes yes yes 12-Nov-99 Voice Communication Switching System Mr. Coyle yes yes (-) yes Mr. Coyle 16-Dec-99 Minuteman III Guidance Replacement System 7-Jan-00 yes yes Mr. Coyle LFT&E of SH-60B/HH-60H 18-Jan-00 yes yes yes yes yes Mr. Coyle Rolling Airframe Missile 3-Apr-00 F/A-18 E/F Super Hornet yes yes yes yes yes Mr. Coyle 23-May-00 SLAM-ER yes yes yes yes yes Mr. Coyle 16-Aug-00 JSTARS Common Ground Station yes yes (-) no Mr. Coyle 1-Oct-00 yes yes(-) Mr. Coyle LFT&E of XM1001 - 40mm Canister Cartridge 17-Nov-00 V-22 OSPREY yes yes no yes yes Mr. Coyle 30-Nov-00 F-15E Tactical EW System, AN/ALQ-135 Band 1.5 yes yes no Mr. Coyle 1-Dec-00 LFT&E of MH-47E/MH-60K Special Operations Aircraft yes yes Mr. Coyle 22-Dec-00 USS OSPREY Class Coastal Minehunter (MHC 51) yes yes yes yes yes Mr. Coyle 18-Jan-01 LFT&E of B-2 Bomber Conventional Warfare Mission no yes (-) Mr. Coyle 2-Mar-01 Joint Direct Attack Munition (JDAM) - 2000 lbs yes yes no Mr. Frame 6-Apr-01 M2A3 Bradley Fighting Vehicle System yes yes yes yes yes Mr. Frame 26-Sep-01 Minuteman III Ppropulsion Replacement System yes yes yes Mr. Frame Exit Criteria Met = yes; Stealth = yes; Avionics = no; Weapons Delivery = no 14-Sep-01 F-22 Test Adequacy Report to Congress (Section 8124) Mr. Christie 3-Oct-01 Predator Medium Altitude Endurance UAV yes no no Mr. Christie 14-Nov-01 Joint Primary Aircraft Training System: T-6A Aircraft yes yes no Mr. Christie 1-Feb-02 Cooperative Engagement Capability (CEC) yes yes yes Mr. Christie 25-Mar-02 Sensor Fuzed Weapon (SFW P3I) yes yes Mr. Christie 28-Mar-02 Advanced Medium Range Air-to-Air (AMRAAM) yes yes Mr. Christie 9-Apr-02 MLRS M270A1 Launcher yes yes yes Mr. Christie 2-Aug-02 MH-60S Fleet Combat Support Helicopter yes yes no yes yes Mr. Christie Classified System 31-Oct-02 PAC-3 Block 2002 System Mr. Christie 23-Dec-02 Shadow 200 Tactical UAV marginal yes(-) no Mr. Christie 3-Feb-03 yes yes Mr. Christie M329E3 Armor Piercing Fin-Stabilized Discarding Sabot-Tracer 9-Apr-03 B-1B Block E Conventional Mission Upgrade Program yes no yes yes yes Mr. Christie 15-Jul-03 Seawolf Nuclear Attack Submarine yes yes yes yes yes Mr. Christie 24-Sep-03 Multifunctional Information Distribution System (MIDS) Mr. Christie Integrated Defensive Electronic Countermeasures Suite (IDECMS) Mr. Christie yes yes no 2-Dec-03 yes yes yes Final Reports: 90% Effective 80% Suitable But a downward trend.

10. DOT&E scores are “final scores” • It is not unusual for a system to do poorly in its IOT&E. • Different Services have, in the past, handled this differently. • Often the effect is to rerun the OT until the test results are “good enough.”

11. 1985-1990 1996-2000 Source: ATEC/AEC

12. Handicappers Guide: Ten ways to tell a program is in trouble without doing a single test • Live fire testing is an issue from outset • Unrealistic requirements • Tests start getting cut long before testing has begun • Program doesn’t want to do M&S early • Program wants to replace testing with M&S

13. - - From the achieves • Program does tests that, if they fail, will make headlines and does it without adequate prep. • Start cutting training manuals, logistic & sustainability • Program is cutting LRIP article and increasing overall production • No money budgeted for second test if first test fails • Program talks to testers only when it is required, and restricts access as a matter of attitude

14. DOT&E also gets scored • In the press • In Congress

15. Washington Post… 1987

20. 2001

21. 1997

22. Interesting Plays • Early involvement • Modeling & simulation • Acquisition Plays • Central T&E Investment Program • Defense Test Resource Management Center

23. Win & Loose is the wrong paradigm • Who is the opponent? • Win – Win is the goal • “business has to make a profit and the soldier has to get what he needs” (~1987) • “products that satisfy user needs with measurable improvements to mission capability and operational support, in a timely manner, and at a fair and reasonable price” (DoD Directive 50001, May 2003)