ASW METOC Metrics: Proposal for VS07 Data Collection

1. ASW METOC Metrics: Proposal for VS07 Data Collection Bruce Ford Clear Science, Inc. (CSI) bruce@clearscienceinc.com Tom Murphree Naval Postgraduate School (NPS) murphree@nps.edu Paul Vodola, Matt McNamara, and Luke Piepkorn Systems Planning and Analysis (SPA) pvodola@spa.com Brief for ASW METOC Metrics Symposium Two 02-04 May, 2007 Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

2. VS07 Opportunity • VS07 presents excellent opportunity to develop, test, and improve data collection and analysis process from multiple perspectives: • NOAT • RBC • MPRA • Exercise • Two key types of data to collect • Performance data • Impacts data • Process data Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

3. Hierarchy of Metrics Higher Level (Navy-wide SLD accuracy) Metric Symposium Focus Space CNMOC/FleetMetrics Larger Spatial and/or Temporal Scale (Exercise forecast location) Directorate Metrics NOAC Metrics Performance(Temperature and salinity accuracy) Impacts (Number of positively identified submarines) Exercise Metrics NOAT Metrics Ind. Forecast Metrics Smaller Spatial and/or Temporal Scale (Point forecast location) Lower Level (MOATs SLD accuracy) Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

4. VS07 Potential Data Collection Sites • Stationing of Data Collectors in Priority Order • Aboard each CV/ASWC NOAT • At RBC • Embedded with CTF-74 • Embedded with deployed MPRA TSC (MOCC) • Embedded with 3rd Fleet staff Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

5. General Data Collection Plan • Aboard each CV • Mimic NOAT data collection role by collecting all: • Discrete data • Verifying in situ data • Recommendations (tactical and mitigation) • Recommendation outcomes • Customer measures of success • Investigate other potential data sources, methods, customer measures of success, etc. • At RBC • Monitor watch officer actions and log for potential data collection • Monitor NOAT interactions • Observe product generation process • Embedded with deployed MPRA TSC (MOCC) • Mimic MPRA data collection by collecting: • Data from each GREEN and PURPLE • Discrete forecast elements • Investigate other potential data sources, methods, customer measures of success, etc. Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

6. General Data Collection Plan (continued) • Embedded with each major staff • Mimic NOAT data collection role by collecting all: • Recommendations (tactical and mitigation) • Recommendation outcomes • Customer measures of success • Investigate other potential data sources, methods, customer measures of success, etc. Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

7. General Data Collection Methods • Each data collector will: • Be thoroughly trained on data collection objectives prior to VS07 (e.g., fast cruise, July 2007) • Record data on specially-prepared paper forms and spreadsheets • Maintain contact with other data collectors via twice-daily chats (hosted by the RBC) • Submit daily spreadsheets to RBC for archival • Document the METOC support process used • Document collection process used • Archive chats • Stay vigilant regarding the overall goals and methods • Record observations and suggestions • Recommend improvements to collection forms/process • Assess how the metrics interface and process will impact future deployed NOATs • Learn more about the METOC support process • Strive to understand the ASW warfighter and how success is measured • Ask questions! Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

8. Who Should be Collectors • Knowledgeable • For instance, those riding carriers will benefit from carrier experience • Knowledge of ASW operations • Innovative thinkers • Collectors will be armed with an overall knowledge of the data collection goals • Creative means of obtaining data is encouraged • In addition: • Must be ready to engage strangers (ASWC personnel) in conversation • Must observe operations as a neutral 3rd party and pay attention to the process • Must record copious thoughts and observations and prepare a detailed report prior to FINEX • Must possess a positive, “can-do” attitude! Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

9. Prior Arrangement for VS07 Metrics Data Collection • Collectors identified (knowledgeable, innovative thinkers!) • Ship riders cleared by prospective host units • Spreadsheets and forms prepared • Training conducted for collectors (NOAT-SSC fast cruise/VTC/visits/DVD) • IRC prepared • Labor and travel cost determined and funded Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

10. Post VS07 Activities • Gather data collectors to discuss findings • Issue data collection report of key findings to ASW metrics executive committee • Propose changes to ASW metrics data collection plan • Collate data and compute metrics (NPS student?) • Investigate potential of sanitizing data for ease of analysis • Assemble exercise level data • Evaluate exercise as a whole • Incorporate data collection and process information in operational modeling effort • Document our process, successes, and failures -– capture lessons learned Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com

11. What It Will Take To “Win” at VS07 • Unwavering top level support – All levels must know that this program is supported • Strong leadership at support nodes supporting upper level goals • Involve METOC customers in the metrics process • Let them know of our efforts • Enlist their support where possible to improve • METOC products and support • Metrics data collection • Share our results Ford B and T. Murphree, ASW Metrics Proposal, May 07, bruce@clearscienceinc.com, murphree@nps.edu