Earth System Modeling Framework

1. Earth System Modeling Framework Cecelia DeLuca, Don Stark, Chris Hill Arctic System Model Workshop May 20, 2008

2. Outline • About ESMF • Best Practices, On and Off List • Governance • Personal History • Conclusion

3. About ESMF • ESMF provides component wrapperswith standard interfaces, a set of data structures for data exchanges, and customizable drivers. • ESMF provides common utilities, such as data communications, regridding, time management, configuration, and message logging. • Goals are component interoperability and software reuse.

4. ESMF Application Example GEOS-5 Atmospheric General Circulation Model Application Example • Each box is an ESMF component • ESMF State objects carry data between components • Every component (including couplers) has a standard interface to facilitate exchanges • The ESMF architecture enables the assembly of many different systems

5. Standardization of Component APIs • There are only three ESMF component methods: Initialize, Run, and Finalize (I/R/F) • Users create a Component by assigning their user code I/R/F methods to an ESMF Component type • The ESMF Component calls back into the specific user-assigned methods • I/R/F methods cascade down the tree • Small set of standard arguments: call ESMF_CompRun (myComp, importState, exportState, clock, phase, blockingFlag, rc)

6. Standardization of Data Structures • ESMF State data structures contain all data exchanged between components • Flexibility in data representation - States can contain lists of varied data structures, including Arrays, ArrayBundles, Fields, FieldBundles, and other States The ESMF philosophy is to constrain the number of component methods and their arguments, but retain flexibility in the data structures used in inter-component exchanges

7. ESMF Adoption - PARSSSE Prepare user code • Split user code into initialize, run and finalize methods and decide on components and coupling fields and control flow Adapt data structures • Wrap native data structures either as • ESMF Arrays (represent data in index space only)Regrid via user-supplied weights + sparse matrix multiply • ESMF Field objects (Arrays + coordinates and metadata) Regrid via on-line weight generation Pack Arrays or Fields into States.Wrap time information in ESMF Clocks. Register user methods • Attach user code initialize, run and finalize methods to ESMF Components through a registration process Schedule, Synchronize, and Send data between components • Write couplers using ESMF redistribution, sparse matrix multiply, regridding, or user-specified transformations Execute the application • Run components using an ESMF driver

8. HAF GAIM ESMF Earth Science Components & Models

9. ESMF Timeline • First phase started in 2002 with NASA funding • Second phase began in 2005 with a transition to multi-agency support and management – NASA, NOAA, NSF, and DoD sponsors • ESMF component interfaces are the basis of programs at multiple agencies: • DoD Battlespace Environments Institute (BEI) • NOAA National Environmental Modeling System (NEMS) • NASA Modeling Analysis and Prediction Program for Climate Variability and Change (MAP) • Community Sediment Transport Model (ONR), others… • In operational use since August 2006, initially at the National Weather Service, ongoing transition of other ESMF coupled systems to operations • Scope of current work will be completed in 2010 • Ongoing diversified funding expected

10. Release Path 2002 2003 2004 2005 2006 2007 2008 2009 2010 ESMF v1 Prototype Full system prototype ESMF v2 Components, VM and Utils ESMF_GridCompRun() Building bottom-up ESMF v3 Index Space Operations ESMF_ArraySparseMatMul() ESMF v4 Grid Operations ESMF_GridCreate() ESMF_FieldRegrid() ESMFv5 Standardization Build, init, data types, error handling, … Standardization Public release ESMF v3.1.0r

11. ESMF Package Description, the Basics • Open source, open development (browsable repository) • Components can run concurrently, sequentially or in mixed mode • Serial or parallel • Single or multiple executable or combinations • Shared or distributed memory or hybrid • Support for model ensembles, including execution of multiple instances in the same address space • 2000+ unit tests, system tests, and examples regression tested nightly on 26+ platform/compiler combinations • Reference Manual, Users Guide and the examples therein updated automatically with code changes

12. ESMF Package Description, Coupling • Data transformations can be executed within a coupler component, or arranged in a coupler component and executed directly between model components • Coupling can be done in index space or physical space • Performance: < 5% overhead in time to solution vs customized native approaches, highly scalable in performance and memory (performance reports online)

13. Upcoming Releases Public Release 3.1.0r, May 2008 • Main capabilities added since last public release - 2.2.2rp3, December 2006: • General grid and data representation and parallel regridding support through an Array index space layer (user supplied interpolation weights) – logically rectangular grids, multi-patch grids, unstructured grids • Representation of rectilinear and curvilinear grids in a Grid class (preparation for framework-generated interpolation weights) • New ports including gfortran and Solaris • Compact, simpler interface: cut number of public data handling methods in half (304 -> 147, with caveats)

14. Upcoming Releases (cont.) Internal Release 3.1.1, June 2008 • Initial implementation of observational data streams • Generation of regridding interpolation weights for logically rectangular grids, bilinear and higher order method • Attribute class can store standardized metadata packets and represent metadata hierarchies - e.g., State metadata includes the metadata of Fields that are stored in it • Initial implementation of unstructured grid interface Patch Release 3.1.0rp*, Summer 2008 • Petascale optimizations for Array and ArrayBundle sparse matrix multiply

15. Relationship to CCSM • Passed CCSM Stage 1 evaluation of ESMF in May 2007, showing < 5% overhead, comparable memory use, no change to component internals. • Result: creation of cpl7 based on ESMF design • Currently working on Stage 2 evaluation:Creation of an ESMF version of cpl7 called ESMCoupler • Expected advantages: • Greater interoperability with other components • More flexible architecture that doesn’t rely on a single central coupler • On-line regridding • Integrated handling of metadata

16. Team Composition Team composition (blue is off-site) • Manager (agency coordination and technical coordination) • Operations Manager (website, metrics, space issues, local administration) • Integrator/Test lead (regression testing, release management) • Tester for numerical methods 7 developers: • Systems level/porting/build • Low level data structures and architecture • Structured grids • Meshes • High level numerical data structures • Language interfaces • Utilities including calendaring • Metadata and attributes External and related • External: ½ FTE performance testing • Related: FTE for component distribution portal

17. Distributed Development GOAL - Everybody on the team has access to all information, current and past • Archived email list where all development correspondence gets cc’d • Frequent telecons with minutes • Web browsable repository, mail summary on check-ins • Daily archived test results • Monthly archived metrics • Public archived trackers (bugs, feature requests, support requests, etc.)

18. Multiple timescales, all staff levels • Places for structured argument ESMF Governance ExecutiveManagement Executive Board Strategic Direction Organizational Changes Board Appointments annually Reporting Interagency Working Group Stakeholder Liaison Programmatic Assessment & Feedback Advisory Board External Projects Coordination General Guidance & Evaluation Reporting Working Project Change Review Board Development Priorities Release Review & Approval Joint Specification Team Requirements Definition Design and Code Reviews External Code Contributions quarterly Functionality Change Requests weekly Resource Constraints Implementation Schedule Collaborative Design Beta Testing Core Development Team Project Management Software Development Testing & Maintenance Distribution & User Support daily

19. Values • Community driven development and community ownership • Openness of project processes, management, code and information • Correctness • Commitment to a globally distributed and diverse development and customer base • Simplicity • Efficiency • Public storage of project records and other information • Engagement Web link for detail: http://www.esmf.ucar.edu/about_us/values.shtml

20. How to Get Started & Get Help • Web meeting with the development team • Support list – esmf_support@ucar.edu • Code Examples page and FAQ • Tutorials, on-line and in person • Annual Meeting (Fort Lauderdale, FL May 28-30, 2008) • Reference Manual and Users Guide

21. Governance • Management of ESMF requires governance that recognizes social and cultural factors as well as technical factors • Main objectives of governance: • Enabling people to argue in a civilized, contained, constructive way • Enabling people to set priorities and make decisions based on resource realities • Coordinating at many levels, from developer to center director to program manager

22. Facilitating Science:Coupled Climate-Chemistry with ESMF The image shows results from a version of the GEOS-5 atmospheric general circulation model coupled to a stratospheric chemistry package (STRAT-CHEM), also developed at NASA, but independently of GEOS-5,which has now been made ESMF compliant.