Development of a Plume-in-Grid Version of Global-through-Urban WRF/Chem

1. Development of a Plume-in-Grid Version of Global-through-Urban WRF/Chem Prakash Karamchandani, Krish Vijayaraghavan, Shu-Yun Chen ENVIRON International Corporation, Novato, CA Yang Zhang, North Carolina State University, Raleigh, NC 9th Annual CMAS Conference, October 11-13, 2010 Chapel Hill, North Carolina

2. Outline • Objectives • Global-through-Urban WRF/Chem • Modeling Domains • Challenges in Coupling Plume Model with WRF/Chem • Solutions • Current Status

3. Objectives • Develop unified Global-through-Urban WRF/Chem (GU-WRF/Chem) for integrated climate change-air quality modeling at all scales • Apply over multiple nested grids with domains covering the globe, the Trans-Pacific region, the continental US (CONUS), and the eastern U.S. • This work: incorporate Plume-in-Grid (PinG) treatment to resolve point source plumes in the eastern U.S., based on the treatment used in CMAQ-APT (CMAQ with Advanced Plume Treatment)

4. GU-WRF/Chem • Extension of mesoscale WRF/Chem to larger scales • Includes CB05 chemistry mechanism with global extensions (CB05GE) to represent urban, regional and remote tropospheric chemistry, and the chemistry of the lower stratosphere • Multiple choices for aerosol treatment based on MADE/SORGAM (modal), MADRID (sectional) and MOSAIC (sectional) • Embedded SCICHEM reactive plume model for PinG treatment (ongoing work)

5. Modeling Domains

6. Plume-in-Grid Implementation Approach • Use plume model in CMAQ-APT as a starting point • Update model to make chemistry (gas, aerosols, aqueous) consistent with that in GU-WRF/Chem • Write new interfaces to couple GU-WRF/Chem with plume model • Conduct initial testing with tracer version of model running in serial mode to verify that plumes are transported correctly within WRF • Test chemistry version of the model in serial mode • Implement parallelization • Conduct summer episode simulation, evaluate model, and assess PinG impacts

7. Challenges • Architectures of CMAQ and WRF/Chem are significantly different • WRF employs a layered software architecture: driver layer, mediation layer, model layer, and external libraries for time-keeping, parallelization interfaces, etc. • WRF architecture promotes modularity, portability and software reuse; however – software design is not very intuitive for physical scientists • Science code developers are generally restricted to the model layer and lower levels of mediation layer • Design works well when you don’t need to go beyond these layers, e.g., for “column physics” or “column chemistry”. For fully 3-D packages, such as PinG, science code developers need to venture to the higher layers of the system

8. Key Features of WRF Relevant to New Science Code Development • Makes extensive use of modern programming features of Fortran 90, such as modules, derived data-types, dynamic memory allocation, pointers, and recursion • The most important derived data type is “domain”. Instances of this data type are used to represent the modeling grids in a WRF or WRF/Chem simulation • This data type contains all the “state” variables associated with a grid, such as meteorology, chemical species concentrations, emissions, photolysis rates, as well as other grid characteristics, such as dimensions and pointers to nested, sibling and parent grids • The subroutine that performs integration of the physical and chemical processes for a single time step is a recursive subroutine, i.e., it is first called for the mother grid and then calls itself for each subsequent nested grid and their siblings until the last grid is integrated

9. WRF Registry • Text file that specifies model state arrays, I/O, coupling, inter-process communication, physics and chemistry packages, and model configuration options • Used at model build time to auto-generate several thousand lines of code, e.g.: • Declaration, allocation, and initialization of state data • Generation of dummy argument lists and their declarations • Generation of actual argument lists used when passing state data between subroutines at the interface between layers in the WRF software architecture • If new science code requires access to WRF variables that are not currently saved as state variables, then the registry needs to be modified by the code developer; similar requirement for model configuration options relevant for the new code

10. Other Considerations • Array conventions (IKJ instead of the traditional IJK) for state variables and automatic arrays • Grid staggering, e.g., horizontal wind components are specified at dot points in CMAQ and cell faces in WRF • Domain dimensions for all state variables are staggered: for variables that are defined at cross or mass points, such as chemical concentrations, this means that there is an extra grid cell at the end in each dimension • All time-keeping information (such as current time, process time step, etc.) is stored in the “domain clock” associated with each grid and needs to be obtained from the external time-keeping library (ESMF) using “wrapper” accessor functions • The model namelist file (which provides run control information for a WRF/Chem simulation) should be updated to select new model configuration options specified in the registry

11. Parallelization in WRF Logical domain 1 Patch, divided into multiple tiles • 2-level domain decomposition: “Patches”, assigned to processes, and “Tiles”, assigned to threads • Model layer routines (science code) generally operate at tile level • Driver layer allocates, stores, and decomposes model domains and provides top-level access to communications and I/O using external libraries • Mediation layer provides interface between driver layer and model layer; performs many important functions, one of which is to dereference driver layer data objects (e.g., grids) and pass the individual fields (state variables) to the model layer routines Inter-processor communication Figure from WRF presentation by John Michalakes, NCAR

12. Implications for PinG Implementation • Domain decomposition paradigm not compatible with SCICHEM • SCICHEM uses puff decomposition for the following reasons: • SCICHEM state variables are puff variables, not grid variables • Load balancing (with domain decomposition, non-uniform distribution of point sources across model domain will increase the load on some processors while other processors would remain idle or perform less work) • Difficulties with treating puffs crossing sub-domains during a time step if domain decomposition is used • Grid state variables used in SCICHEM to provide 3-D meteorology and chemistry inputs: information required for whole domain • Model interface needs to collect sub-domain state variable information (inter-processor communication required)

13. PinG in GU-WRF/Chem • Main interface in the mediation layer • All WRF grid description and meteorology state variables de-referenced in interface and passed as simple arrays to SCICHEM driver (in CMAQ-APT these variables were read from I/O API data files) • WRF/Chem grid chemistry variables de-referenced in interface and passed as simple array to SCICHEM driver (similar to treatment in CMAQ-APT) • New state variables specified in Registry (e.g., dry deposition velocities), as well as new configuration options for PinG treatment • For parallel implementation, write code to gather full domain state arrays at beginning of SCICHEM time step and scatter chemistry variables at end of time step

14. Current Status • Meteorology and grid interfaces completed • Tracer version of model currently being tested • Updates to SCICHEM chemistry • Development of chemistry interfaces • Parallelization

15. Acknowledgments This work is being conducted under subcontract to North Carolina State University for EPA STAR Grant # RD833376

16. Questions?