Introduction to the Earth System Modeling Framework

1. Introduction to the Earth System Modeling Framework NASA GSFC PSAS Climate MITgcm GFDL FMS Suite Weather NCAR/LANL CCSM NCEP Forecast Data Assimilation NSIPP Seasonal Forecast C. DeLuca/NCAR, J. Anderson/NCAR, V. Balaji/GFDL, B. Boville/NCAR, N. Collins/NCAR, T. Craig/NCAR, C. Cruz/GSFC, A. da Silva/GSFC, R. Hallberg/GFDL, C. Hill/MIT, M. Iredell/NCEP, R. Jacob/ANL, P. Jones/LANL, B. Kauffman/NCAR, J. Larson/ANL, J. Michalakes/NCAR, E. Schwab/NCAR, S. Smithline/GFDL, Q. Stout/U Mich, M. Suarez/GSFC, A. Trayanov/GSFC, S. Vasquez/NCAR, J. Wolfe/NCAR, W. Yang/NCEP, M. Young/NCEP and L. Zaslavsky/GSFC

2. Outline • ESMF Overview • ESMF Applications • Related Projects • ESMF Architecture • Timeline

3. Background NASA’s Earth Science Technology Office proposed the creation of anEarth System Modeling Framework (ESMF) in the September 2000 NASA Cooperative Agreement Notice: “Increasing Interoperability and Performance of Grand Challenge Applications in the Earth, Space, Life and Microgravity Sciences” A large, interagency collaboration with roots in the Common Modeling Infrastructure Working Group proposed three interlinked projects to develop and deploy the ESMF, which were all funded: Part I: Core ESMF Development (PI: Killeen, NCAR) Part II: Modeling Applications (PI: Marshall, MIT) Part III: Data Assimilation Applications (PI: da Silva, NASA GMAO)

4. Motivation In climate research and NWP... increased emphasis on detailed representation of individual physical processes; requires many teams of specialists to contribute components to an overall modeling system In computing technology... increase in hardware and software complexity in high-performance computing, as we shift toward the use of scalable computing architectures In software …development of frameworks, such as FMS, GEMS, CCA and WRF, that encourage software reuse and interoperability The ESMF is a focused community effort to tame the complexity of models and the computing environment. It leverages, unifies and extends existing software frameworks, creating new opportunities for scientific contribution and collaboration.

5. ESMF Project Description GOALS: To increase software reuse, interoperability, ease of use and performance portability in climate, weather, and data assimilation applications PRODUCTS: • Core framework: Software for coupling geophysical components and utilities for building components • Applications: Deployment of the ESMF in 15 of the nation’s leading climate and weather models, assembly of 8 new science-motivated applications METRICS: RESOURCES and TIMELINE: $9.8M over 3 years

6. ESMF Accelerates Advances in Earth System Science Eliminates software barriers to collaboration among organizations • Easy exchange of model components accelerates progress in NWP and climate modeling • Independently developed models and data assimilation methods can be combined and tested • Coupled model development becomes a truly distributed process • Advances from smaller academic groups easily adopted by large modeling centers Facilitates development of new interdisciplinary collaborations • Simplifies extension of climate models to upper atmosphere • Accelerates inclusion of advanced biogeochemical components into climate models • Develops clear path for many other communities to use, improve, and extend climate models • Many new model components gain easy access to power of data assimilation

7. ESMF Collaborators NOAA GFDLAnts LeetmaaV. BalajiRobert HallbergShep Smithline NASA GMAOArlindo da Silva, PIMichele RieneckerMax SuarezAtanas TrayanovChristian KeppenneLeonid ZaslavskyWill SawyerCarlos Cruz NOAA NCEPStephen LordMark IredellMike YoungWeiyu YangJohn Derber MITJohn Marshall, PIChris Hill DOE LANL Phil Jones University of MichiganQuentin Stout NSF NCARTim Killeen, PIJeff AndersonByron BovilleNancy Collins Cecelia DeLucaRoberta JohnsonAl KellieJohn MichalakesDavid NeckelsEarl Schwab Robbie StauferSilverio Vasquez Jon WolfeDOE ANLRob JacobJay Larson

8. Outline • ESMF Overview • ESMF Applications • Related Projects • ESMF Architecture • Timeline

9. Modeling Applications

10. Data Assimilation Applications

11. Interoperability Experiments:8 New Applications

12. Outline • ESMF Overview • ESMF Applications • Related Projects • ESMF Architecture • Timeline

13. Related Projects • CCA is creating a minimal interface and sets of tools for linking high performance components. CCA can be used to implement frameworks and standards developed in specific domains (such as ESMF). • Collaborators include LANL, ANL, LLNL, ORNL, Sandia, University of Tennessee, and many more. Ongoing ESMF collaboration with CCA/LANL on language interoperability. • Working prototype demonstrating CCA/ESMF interoperability, to be presented at SC2003. • PRISM is an ongoing European Earth system modeling infrastructure project • Involves current state-of-the-art atmosphere, ocean, sea-ice, atmospheric chemistry, land-surface and ocean-biogeochemistry models • 22 partners: leading climate researchers and computer vendors, includes MPI, KNMI, UK Met Office, CERFACS, ECMWF, DMI • ESMF is working with PRISM to mergeframeworks and develop common conventions For joint use with PRISM, ESMF developed a component database to store component import/export fields and component descriptions

14. Larson/ANL DeLuca/NCAR-SCD Jones/LANL Stout/U Mich Killeen/NCAR Drake/ORNL Boville/NCAR-CGD Michalakes/NCAR-MMM Suarez/NASA Goddard Balaji/NOAA GFDL ESMF Connections ESMF Earth System Modeling Framework CCA DOE Common Component Architecture SciDAC DOE/NSF CCSM SciDAC Project GEMS Goddard Earth Modeling System FMS GFDL Flexible Modeling System SWMFSpace Weather Modeling Framework WRF Weather Research and Forecast Model CCSMCommunity Climate System Model PRISM Program for Int. Earth System Modeling CCA SciDAC PRISM WRF CCSM ESMF FMS GEMS SWMF

15. Outline • ESMF Overview • ESMF Applications • Related Projects • ESMF Architecture • Timeline

16. Computational Characteristicsof Weather/Climate Platforms • Mix of global transforms and local communications • Load balancing for diurnal cycle, event (e.g. storm) tracking • Applications typically require 10s of GFLOPS, 100s of PEs – but can go to 10s of TFLOPS, 1000s of PEs • Required Unix/Linux platforms span laptop to Earth Simulator • Multi-component applications: component hierarchies, ensembles, and exchanges • Data and grid transformations between components • Applications may be MPMD/SPMD, concurrent/sequential, combinations • Parallelization via MPI, OpenMP, shmem, combinations • Large applications (typically 100,000+ lines of source code) Seasonal Forecast coupler ocean assim_atm sea ice assim atmland atm land physics dycore

17. Architecture Components Layer: Gridded Components Coupler Components ESMF Superstructure • ESMF provides an environment for assembling geophysical components into applications. • ESMF provides a toolkit that components use to • increase interoperability • improve performance portability • abstract common services User Code Model Layer ESMF Infrastructure Fields and Grids Layer Low Level Utilities External Libraries BLAS, MPI, NetCDF, … • Becoming an ESMF Component • Pack model import and export data into ESMF data structures and conform to a standard calendar. Use ESMF utilities internally as desired. Organize model using standard ESMF methods: Initialize, Run, Finalize, ReadRestart, WriteRestart. Methods may be multi-phase (Run phase=1, Run phase=2). Method interfaces are prescribed. • Instantiate an ESMF Component with name, type, config information. Register standard model methods with Component. If desired, register data. • Use ESMF AppDriver to sequence and run Components.

18. Design Features ESMF enables modeling applications to be: • ScalableModels are built from modular components, and can be easily nested within larger applications • Performance - portableESMF high-performance communication libraries offer a consistent interface across computer architectures • ExchangeableStandard component interfaces enable interoperability

19. ESMF Class Structure GridComp Land, ocean, atm, … model CplComp Xfers between GridComps State Data imported or exported Superstructure Infrastructure Bundle Collection of fields Regrid Computes interp weights Field Physical field, e.g. pressure Grid LogRect, Unstruct, etc. PhysGrid Math description DistGrid Grid decomposition F90 Array Hybrid F90/C++ arrays DELayout Communications Route Stores comm paths C++ Utilities Machine, TimeMgr, LogErr, I/O, Config, Base etc. Data Communications

20. Design Principle:Local Communication All inter-Component communication within ESMF is local - all communication is handled within Components. This allows the architecture of the framework to be independent of the communication strategy. climate_comp As a consequence, Coupler Components must be defined on the union of the PEs of all the Components that they couple. In this example, in order to send data from the atmosphere Component to the ocean, the atm2ocn Coupler mediates the send. atm2ocn _coupler atm_comp ocn_comp phys2dyn_coupler atm_phys atm_dyn PE

21. Design Principle:Scalable Applications Since each ESMF application is also a Component, entire ESMF applications can be treated as Gridded Components and nested within larger applications. climate_comp ocn2atm_coupler Example: atmospheric application itself composed of multiple Components may be run standalone, or nested within a larger climate application ocn_comp atm_comp phys2dyn_coupler atm_phys atm_dyn PE

22. Design Principle: Modularity Gridded Components don’t have access to the internals of other Gridded Components, and don’t store any coupling information. Gridded Components may: • pass their States to other Components through their argument list or • receive user-defined methods through their argument list for transforming and transferring States. These are called Transforms, and they contain a function pointer and attributes describing frequency and validity criteria. Transforms may modify the data in States, receive States from other Components, send States to other Components, etc. EX 1: One-way coupling from atm to ocn atm_xform is a send; ocn_xform a receive EX 2: Standalone atm atm_xform is a no-op ! From climate comp call ESMF_CompRun(atm, atm_xform) call ESMF_CompRun(ocn, ocn_xform) ! From atm comp call ESMF_StateXform(atm_ex_state, & atm_xform) ! From ocn comp call ESMF_StateXform(ocn_im_state, & ocn_xform) ! From atm comp call ESMF_CompRun(ocn, ocn_xform) call ESMF_StateXform(atm_ex_state, & atm_xform)

23. Design Principle:Uniform Communication API The same programming interface is used for shared memory, distributed memory, and combinations thereof. Machine model = abstraction of machine architecture (num_nodes, num_pes_per_node, etc.) DE = a decomposition element - may be virtual, thread, MPI process DELayout = an arrangement of DEs, in which dimensions requiring faster communication may be specified and resources arranged accordingly DELayout: 4 x 3, ESMF_COMM_SHR in x and ESMF_COMM_MP in y The data in a Grid is decomposed according to the number and topology of DEs in the DELayout Performance of ESMF AllGather vs. raw MPI

24. Outline • ESMF Overview • ESMF Applications • Related Projects • ESMF Architecture • Timeline

25. Time Line

26. More Information ESMF website:http://www.esmf.ucar.edu Acknowledgements The ESMF is sponsored by the NASA Goddard Earth Science Technology Office.