Evolution

1. Earth System Modeling InfrastructureCecelia DeLuca and the ESMF TeamNCAR/CISLCCSM Software Engineering Working GroupJune 17, 2009

2. Evolution • Extending beyond historical emphasis on high performance frameworks • Integration across model, data, and organizational elements • Multi-agency and international contributions and partnerships essential

3. Elements of Modeling Infrastructure • Tight coupling tools and interfaces • Emphasis on performance • Frequent, high volume transfers on high performance computers • Examples: ESMF, Flexible Modeling System (FMS) • Loose coupling protocols and interfaces • Emphasis on ease of use • Lower volume and infrequent transfers on desktop and distributed systems • Examples: Open Modeling Interface (OpenMI, hydrology), web services • Science gateways • browse, search, and distribution of model components, models, and datasets • workflows, visualization and analysis services • workspaces and management tools for collaboration • Examples: Earth System Grid (ESG), Hydrologic Information System (HIS)

4. Elements of Modeling Infrastructure • Metadata conventions and ontologies • Structured information describing modeling artifacts • Ideally, with automated production of metadata from models • Examples: CF conventions, CIM (from EU Metafor project) in CMIP5 • Governance • Coordinated and controlled evolution of systems • Example: ESMF Change Review Board – quarterly multi-agency meeting that sets priorities and tasks and creates release schedules

5. Sample Applications • Multi-model ensembles for weather and climate • Automated modeling workflows • Meeting stakeholder requests for model output outside the scope of the original research • Coordinating standards development across the international community

6. Programs • National Unified Operational Prediction Capability (NOAA and DoD operational NWP centers) • Global Interoperability Program (NOAA climate and weather) • Modeling Analysis and Prediction Program (NASA) • Battlespace Environments Institute (DoD regional codes) • Common Metadata for Climate Modelling Digital Repositories (METAFOR, IS-ENES)

7. ESMF Update Cecelia DeLuca, Kathy Saint

8. Release Plan 2002 2003 2004 2005 2006 2007 2008 2009 2010 ESMF v1 Prototype ESMF v2 Components, VM and Utils ESMF_GridCompRun() ESMF v3 Index Space Operations ESMF_ArraySparseMatMul() ESMF v4 Grid Operations ESMF_GridCreate() ESMF_FieldRegrid() ESMFv5 Standardization Build, init, data types, error handling, … Beta release May 2009 ESMF v4.0.0

9. Current Release • ESMF parallel regrid: • Grid-Grid, Grid-Mesh, Mesh-Mesh • Bilinear and higher order Beta Release 4.0.0 May 2009 • Basis for next public release • On-line and off-line (file to file) parallel generation of interpolation weights, bilinear and higher order methods • Two higher-level data representations: connected logically rectangular blocks or fully unstructured mesh • Attributes for Metadata handling, read and write in XML or plain text formats • Optimization of sparse matrix multiply for large processor counts (10,000+) The Flow-Following Finite Volume Icosahedral Model (FIM) from NOAA GSD is converting to ESMF to couple to National Weather Service models

10. Performance Portablity • 30+ platform/compiler combinations regression tested nightly (3000+ system and unit tests) • Newer ports include Solaris and native Windows Performance at the petascale… The chart at right shows scaling of the ESMF sparse matrix multiply, used in regridding transformations, out to 16K processors. (ESMF v3.1.0rp2, MCT 2.5.1) Plot from Peggy Li, NASA/JPL Tested on ORNL XT4 in a variety of configurations . -N1 means 1 core per node. ASMM Run-Time Comparison msec

11. Loose coupling andintegration with data services • Prototype completed that translates ESMF initialize/run/finalize component interfaces into web services • Supports loose coupling with hydrological, human impacts, and other services, and integration of components into service oriented architectures • Will enable invocation of ESMF components and applications from a portal in a Grid-based (e.g. Teragrid) gateway • Data and metadata generated by ESMF Attributes during runs can be stored back to portals for search, browse, comparison, and decision support Portal developed jointly with the Earth System Grid displays detailed scientific metadata … work funded by NSF.

12. Web services: design goals • ESMF components can be turned into web services with minimal impact to component developers • Fortran and C++ • Use standard web technologies • SOAP/XML • NetCDF • OpenDAP/Thredds • Simple deployment and user interface • 2 for 1 interface – with ESMF can get both a tight coupling interface and a service interface

13. Service Architecture Client Application WSDL SOAP Client Internet WSDL Tomcat/ Axis2 SOAP Service OpenDAP Component Connector Component List Application Registrar libnetesmf NetCDF Grid/Coupler Component

14. Current Status • Initial prototype released • Grid Component wrapper w/ SOAP interface • Initialize, Run, Finalize component operations • Current version • Support for Coupler components • Single SOAP service to support multiple components • Data movement supported using OpenDAP access to NetCDF files • Next steps • New project to integrate with hydrology data and applications, including Hydrologic Information System (HIS) Desktop

15. ESMF in CCSM4 Fei Liu, Bob Oehmke, Ufuk Turuncoglu

16. ESMF in CCSM4 • Motivation: create an ESMF based CCSM4 platform to explore: • Self-describing models • Interoperability with other modeling systems • Component as web service • Automated model workflow • Online regridding • Improved scalability • General Approach • Introduce a wrapper layer between CCSM4 driver and ESMF component • Convert MCT-based model component to ESMF-based component • Verify against baseline global integrals to ensure they are bit-wise reproducing. • Completed all of CCSM4 dead, data and active components

17. Plan for CCSM4 ESMF Library MCT based CCSM Driver ESMFshr code Shared code MCT -> ESMF_Array Domain -> ESMF_ArrayBundle Infodata -> ESMF_State Attributes ESMF_Array -> MCT ESMF_ArrayBundle -> Domain ESMF_State Attributes -> infodata MCT based model component ESMF compliant model component Data structure independent model physics

18. Regridding Bob Oehmke

19. Higher-Order Interpolation • CCSM now uses ESMF higher-order interpolation as its standard non-conservative remapping • Replaces bilinear interpolation of atmosphere to ocean states • Results in a large reduction in noise in ocean surface transports (33% in a measured case) • Higher-order interpolation: • A patch is a 2nd order n-D polynomial representing source data • Patches generated for each corner of source cell • Each patch created by least-square fit through source data in cells surrounding corner • Destination value is weighted average of patch values • Full algorithm presented here in 2007 by David Neckels • Longer description in ESMF v4.0.0 Reference Manual

20. Weight Generation • Higher-order weights generated by ESMF off-line application • Takes netCDF grid files and generates netCDF weight file • Format same as SCRIP, can be used as an alternative • Runs in parallel • Multiple improvements over last year • Improved accuracy • More robust • Pole options are now: no pole, full-circle average, and n-point stencil pole • Future work • More pole options (improved vector quantity handling) • Weight generation for conservative regridding

21. Noise reduction in CCSM transport • ESMF higher order interpolation weights were used to map from a 2-degree Community Atmospheric Model (CAM) grid to an irregularly spaced POP ocean grid (384x320) • dTAUx/dy was computed using interpolated fields – this is closely related to the curl of the wind stress, which drives the upper ocean circulation • Noise is calculated as deviation of a point from the sum of itself plus four neighbors • 33% reduction in noise globally compared to original bilinear interpolation Interpolation noise in a derivative of the zonal wind stress Interp. noise grid index in latitudinal direction Black = bilinear Red = higher-orderESMF v3.1.1 Green = higher order ESMF v4.0.0

22. Attributes in CCSM4 Ufuk Turuncoglu

23. CCSM4 and ESMF Attributes ESMF Attributes hold metadata about model components, fields, etc. Using ESMF Attributes, metadata about CCSM4 driver fields was exported in XML format Metadata includes: name long name (or description) standard name units whether it is an import or export field All the field metadata uses the CF conventions and Metafor Common Information Model (CIM) The standard names of CCSM driver fields were collected and corrected by working with the Curator project and component liaisons.

24. CCSM and ESMF Attributes Example: Contents of output XML file for CCSM atmosphere component;

25. Towards CCSM self-describing workflows Ufuk Utku Turuncoglu1 Sylvia Murphy2 Cecelia DeLuca2 1 Istanbul Technical University 2 National Center for Atmospheric Research * please come and see our poster for more information

26. Outline Motivation Basic Concepts Implementation of CCSM Workflow Preliminary Results Future Plans

27. Motivation • Create easy to use work environments for running complex models like CCSM4 • Automatically document parameter changes and other aspects of model runs • Hide interactions with complicated computing technologies • Archive, reproduce and share experiments

28. Workflows • A modeling workflow separates modeling tasks into smaller chunks, and combines them using dataflow pipelines. • KEPLER was chosen because it is open sourceand supports different models of computation. • A KEPLER director controls the execution of a workflow and sends execution instructions to the actors that are specialized to do sub-tasks. In other words, actors specify “what processing occurs” while the director specifies “when it occurs”.

29. Provenance • Provenance is defined as structured information that keeps track of the origin and derivation of the workflow. • The basic types of provenance information: • System (system environment, OS, CPU architecture, compiler versions etc.) • Data (history or lineage of data, data flows, input and outputs, data transformations) • Process (statistics about workflow run, transferred data size, elapsed time etc.) • Workflow (version, modifications etc.) • Seek to collect provenance in a format that is easily displayed in a portal and linked to data outputs

30. Components of Workflow Environment Typical workflow

31. Hierarchical Collection of Provenance Information Multi-component structure of CCSM makes it complicated to collect provenance information.

32. Conceptual CCSM Workflow • Split into two parts: run preparation and postprocessing • Postprocessing is automically triggered when run workflow completes

33. Kepler Graphical Interface main workflow composite actor CCSM workflow

34. Development Tasks • CCSM modifications for ESMF Attributes (explained earlier) • A Perl script was written to setup the third-party environment on remote machines. • ORNL/NCSU scripts that gather system provenance information were modified to work with CCSM. • Added support for multiple source directories, multi-component models, XML output • Collects extra system provenance information (compiler etc.) • Created new Kepler actors: • Grid; CCSM: build, modify, and run; post-processing • Listener to trigger other workflows automatically • Provenance recorder with modified XML option

35. CCSM Workflow Actors Created KEPLER actors: Grid actors; Job submission to TeraGrid CCSM specific actors; create, modify (env_*.xml) Post-processing actors Utility actors

36. Results & Future Plans A fully active CCSM4 workflow with BTR1 component set and 1.9x2.5_gx1v6 resolution was run on the TeraGrid (IBM BG/L Frost) for five days Provenance information was automatically collected Near future plans: Longer simulation will be done with two different resolutions and two different computers (Frost and Kraken) simultaneously Export collected provenance information into ESG

37. Questions? Contact: turuncu@ucar.edu Project Repository: http://esmfcontrib.cvs.sourceforge.net/viewvc/esmfcontrib/workflow/ Demo Movie: http://www.be.itu.edu.tr/~turuncu/workflow.htm

38. CCSM Metadata forIPCC Assessment Report 5 Sylvia Murphy/NCAR

39. Curator and CMIP5 Curator is coordinating metadata generation, standardization, and implementation in science gateways • We are partners with Metafor who are responsible for the collection of metadata for AR5 • We are collaborating closely with the Earth System Grid (ESG) to implement metadata technologies to search, explore, and compare model information • Curator Partners • National Center for Atmospheric Research • Geophysical Fluid Dynamics Laboratory • Georgia Institute of Technology • EU Partners (BADC, BODC, etc.)

40. ESG data collections

41. CCSM AR4 Metadata

42. End-to-end modeling • Linking datasets and simulations within a science gateway is just one part of an end-to-end modeling system that includes: • Self-describing models • Broader range of coupling interactions • Web services • Workflows • Community-developed ontologies • Science gateways • Future work within Curator will combine these pieces into a variety of modeling systems

43. Questions? For more information please email us at curator@ucar.edu http://curator.ucar.edu