HPCC Best Practices Exchange Report to COPC November 16, 2010

1. HPCC Best Practices ExchangeReport to COPCNovember 16, 2010 Allan DarlingDeputy Director, NCEP Central Operations NOAA NWS NCEPAllan.Darling@noaa.gov

2. HPCC Best Practices - Action Item Recommendation from 2009 Community Review of NCO (UCAR) NCO should actively engage with other similar centers around the world and participate, to the extent possible, in international forums on numerical prediction, high performance computing, and related topics. A key mechanism for both understanding and impacting directions in the international prediction and computing communities is active engagement in professional meetings, exchange visits, and sharing of best practices and tools. Action Item 2010-1.3: Coordinate at least one face-to-face meeting between OPC high performance computing and communications (HPCC) system experts and senior staff to exchange best practices, tools, and processes related to HPCC management, software engineering, and software (model) implementation. 2

3. HPCC Best Practices NCEP managers and contractors travelled to ECMWF and UKMO in August 2010 for two-day best-practices exchange meetings at each center. Best Practices briefings included the following: Intended capability – what is the desired outcome of the practice How practice is accomplished – technical and/or process details Metrics – measuring the effectiveness of the practice Key aspects that set this apart from other implementations of the practice Lessons learned in developing the practice Sustaining and evolving the practice 3

4. HPCC Best Practices - Key Findings Strict adherence to a Development/Production system allocation Maintain development capacity vs. favoring production Typically 3:1 (development : production) Significant effort expended on maximizing utilization of system compute Development work scheduled adjacent to production work and supported the same as production Dedicated code optimization staff with annual performance improvement goals Consume majority of production allocation in the first 6-9 months after an upgrade 4

5. HPCC Best Practices - Key Findings Careful control of transition-to-operations Formalized code handoff from development to operations Dedicated testing and implementation staff independent of developers Limited number of full suite upgrades per year (3-4) with 6-8 week parallel processing prior to each transition Close senior management oversight of T2O Production timeliness sustained with strong failure mitigation capabilities Delayed product dissemination coupled with highly granular jobs Reuse of previous cycle products through relabeling Product generation outside of supercomputers in dedicated systems 5

6. HPCC Best Practices - Key Findings Strong in-house tool development capabilities Custom job schedulers Architecture-specific monitoring tools allowing focused optimization efforts Facilities & System Redundancy Systems kept exactly symmetrical Co-located data centers with in-house management Simplified data exchange Full system & data center independence, including enclosure Extra effort required to protect archive Organization owns data center, leases systems Formalized IT Management ITIL practices mandated Annual & multi-year planning based on industry practice 6

7. HPCC Best Practices – Next Steps Provide trip report to FNMOC and schedule visit Identify topics of interest, e.g: Configuration / Change management Customer outreach Data center management / design Job scheduling System optimization Production management Possible alignment with Spring COPC Coordinate with AFWA Identify topics of interest 7