1. 30 Years of Navy Modeling and Supercomputers: an Anecdotal History Tom Rosmond Marine Meteorology Division Naval Research Laboratory Monterey, California USA

2. Outline • Before my Time: 1961-1974 • My Early Years: 1974-1980 • The Golden Years of NWP: late 1970’s-early 1980’s • Up to the Present • Conclusions and Thoughts for the Future

3. Before my Time: 1961-1974 THE BEGINNING • 1959: Capt Wolff sets up shop on NPS Campus, Monterey • CDC 1604: Seymour Cray’s first design with CDC (first supercomputer?) • 1961: Establishment of Fleet Numerical Weather Facility (FNWF) • 1964: Routine dissemination of numerical products • Successive correction analyses of SLP, upper air heights • NH Barotropic and thickness advection models

4. Before my Time: 1961-1974 CDC 6500’s and NHPE • 1967: FNWC procurement of CDC 6500 to replace 1604 • Dual-processors, comparable to 386 microprocessor in FP performance • Supercomputer of time was Seymour Cray’s CDC 6600 (NMC in 1966) • 1969: Second 6500 acquired • 1970: Northern Hemisphere Primitive Equation (NHPE) model operational on 4 processors. • World’s first multi-processor production code

5. Before my Time: 1961-1974 4-processor 6500 NHPE • Developed by P. Kessel and F. Winninghoff • Processors shared model data through extended core storage (ECS) • OS modifications to allow processor synchronization • Similarities to both shared memory (OpenMP) and distributed memory (MPI) programming models • Parallel efficiency = ~ 75% • An impressive achievement that is certainly underappreciated • An example of the struggle it was to fit models onto these early systems

6. My Early Years: 1974-1980 • 1974: I joined the Environmental Prediction Research Facility (EPRF)1 • Little involvement in FNWC2 model development • Access of FNWC computer systems (6500’s, 1604) • 1976: NEPRF Numerical Modeling Department formed at request of FNWC CO Capt. R. Hughes • He realized that FNWC was unable to maintain R & D continuity needed for NWP system development. • Need for a global NWP capability was major motivation, and he was committed to getting computer system to support it. • 1976: FNWC acquired CDC CYBER 175 (similar to CDC 7600) 1My organization: EPRF NEPRF NOARL NRL-Monterey 2Fleet Numerical: FNWF FNWC FNOC FNMOC

7. My Early Years: 1974-1980 NEPRF Global NWP System Development • 1976: Navy Operational Global Atmospheric Prediction System (NOGAPS) • UCLA General Circulation Model • Barnes Successive Corrections Global Analysis • Variational Initialization with Balance Equation Constraint • 1977: We also had spectral model dynamical core • UCLA GCM physics • Sat on shelf for several years in favor of UCLA GCM based system • 1980: Prototype NOGAPS running on CYBER 175

8. My Early Years: 1974-1980 • 1977: Benchmarking for FNWC procurement • NCAR Cray-1, first system outside Los Alamos? • UCLA GCM 2.40 x 3.00, 6 levels • First introduction to vector programming • Crude compiler + non-vectorizable code = poor performance • Overly conservative target performance, CYBER 203 could compete • Allowed subsequent CDC success with CYBER 205 • 1980: CYBER 203 delivered to FNOC • Heroic work by CDC to get UCLA model to run fast • 1982: CYBER 203 replaced with CYBER 205

9. The Golden Years of NWP1st half: late 70’s – mid 80’s • Supercomputers were cheapest computers you could buy (price/performance) • Cray: Cray-1, XMP, etc • CDC: CYBER 203/205, ETA-10 • IBM: 360-195VP • Development of global NWP forecast systems • Dominance of spectral models • Increased realization of importance of data assimilation • Establishment of ECMWF • “Raised the stakes” in operational NWP • Accelerated progress • Provided gold standard

10. The Golden Years of NWPNavy operational models • NOGAPS • UCLA-GCM - 2.40x3.00 x L9 : 1-pipe/8Mbyte C205 : ~25min/fcstday • Spectral – T47L18 : 2-pipe/32Mbyte C205 : ~6min/fcstday • Spectral – T79L18 : 2-pipe/32Mbyte C205 : ~25min/fcstday (32bit) • Spectral – T79L24 : 4-pipe/64Mbyte C205 : ~12min/fcstday (32bit) • NORAPS (developed by Rich Hodur, NEPRF) • Globally relocatable regional model • 6-8 areas run operationally • Varying resolution, domain and grid sizes • e.g. ran over South Atlantic during Falklands war (1980)

11. Some special comments about the CYBER 205 • CDC aggressively pursued meteorology market, both operations and research • FNOC, NMC, UKMO, GSFC, GFDL, etc • CRAY systems were better general purpose, but 205 excelled on our applications • Initially 205 was difficult to use, compiler/user software was rudimentary, but • Language extension showed how machine worked • Rich array of exotic vector hardware instructions • System software matured • 32 bit/64 bit floating point support • Easy mixing of Fortran and explicit vector instructions, we could get “close to the hardware” • Spectacular percentage of peak performance possible

12. The Golden Years of NWP2nd half: mid 80’s – early 90’s • Introduction of multi-processor systems • Divide and conquer • X/MP, Y/MP, ETA-10 • Multi-tasking/vectorization programming model (parallel/vector) • Spelled the end of single processor supercomputers, e.g. CYBER 205 • Important changes in technology • Price/performance advantage of supercomputers ending • Introduction of desktop workstations • Many people didn’t need supercomputers, just cheap computing • Shrinking supercomputer market

13. Up to the Present:early 1990’s • 1990: Introduction of CRAY C90 • The “best” supercomputer ever? • Run by a higher percentage of NWP operational/research centers than any system before or since. • Parallel/vector programming model very user friendly • Easy to get high percentage of peak performance • 1991: 8 processor C90 at FNMOC • NOGAPS: T159L24 – 6 processors : 10 min/fcstday • 1996: 8 and16 processor C90’s at FNMOC • NOGAPS: T159L24 – 12 processors : 6 min/fcstday • 1997: COAMPSTM replaced NORAPS as Navy’s regional forecast system

14. Up to the PresentMid to Late 1990’s • Beginnings of dramatic changes in supercomputer industry • Scalable commodity based architectures appearing, e.g. CRAY T3E • Powerful workstations replacing supercomputers for many applications • End of domination by American supercomputer vendors • CDC/ETA were long gone • CRAY sold to SGI • MTTB (mean time to bankruptcy) very short for new companies • Rise of Japanese vendors to dominate NWP marketplace, at least outside U.S. • Fujitsu • NEC

15. Up to the Present2000 to today • Proliferation of commodity based, scalable architectures for NWP applications • Many T3E’s still in use • SGI: Origin 2000, Origin 3000 • IBM: SP3, SP4 • Linux clusters are also viable alternatives, especially when price/performance is overriding issue • But, rumors of the demise of vector architectures are exaggerated • 2000: NEC only vendor with traditional vector architecture, SX-6, but • 2002: Resurrected CRAY, Inc introduced X-1 • Price/performance vs ultimate performance is central question concerning long-term prospects of these systems

16. Up to the Present2000 to today • 2001: FNMOC replaced C90’s with O3000’s • 1152 total processors • NOGAPS: T239L30 – 120 processors: 6 min/fcstday • COAMPSTM : 8-11 areas – 40-180 processors: 10-30 min/fcstday • 2003: Operational NAVDAS/NOGAPS (3DVAR): 60 processors • 2004: Direct radiance retrievals with NAVDAS • 2004: Currently under development at NRL, Monterey • T479L54 semi-Lagrangian NOGAPS • NAVDAS-AR: 4DVAR extension of NAVDAS • COAMPSTM/WRF: more areas, higher resolution • Clearly computational requirements are never satisfied!!!

17. Conclusions and thoughts for the future • Vector architectures will continue to be viable candidates for NWP applications • Sharing some features with commodity systems, e.g. caches • When ultimate performance overrides price/performance • Will be part of heterogeneous computing environments • Scientific computing ( and therefore NWP) is now niche market • Be thankful that video game applications share many of our requirements • Consumer based industry has driven some hardware costs to astonishingly low levels, e.g. hard drives • We must work with vendors to ensure that our requirements are not forgotten • Fortran compilers • High-performance interconnects