1 / 62

WRF Model: Software Architecture and Code Structure

WRF Model: Software Architecture and Code Structure. John Michalakes, NCAR. NCAR: W. Skamarock, J. Dudhia, D. Gill, A. Bourgeois, W. Wang, C. Deluca, R. Loft NOAA/NCEP: Tom Black, Jim Purser, S. Gopal NOAA/FSL: T. Henderson, J. Middlecoff, L.Hart U. Oklahoma: M. Xue

benito
Télécharger la présentation

WRF Model: Software Architecture and Code Structure

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WRF Model: Software Architecture and Code Structure John Michalakes, NCAR NCAR: W. Skamarock, J. Dudhia, D. Gill, A. Bourgeois, W. Wang, C. Deluca, R. Loft NOAA/NCEP: Tom Black, Jim Purser, S. Gopal NOAA/FSL: T. Henderson, J. Middlecoff, L.Hart U. Oklahoma: M. Xue AFWA: J. Wegiel, D. McCormick C. Coats (MCNC), J. Schmidt (NRL), V. Balaji (GFDL) S. Chen (UC Davis), J. Edwards (IBM) Acknowledgement: Significant funding for WRF software development from DoD HPCMO CHSSI program (CWO6)

  2. Outline • Software architecture and implementation • Goals and design aspects • Parallelism • Code Structure and Model Layer Interface • Registry • I/O • Extending WRF: Example

  3. Goals Community Model Good performance Portable across a range of architectures Flexible, maintainable, understandable Facilitate code reuse Multiple dynamics/ physics options Run-time configurable Nested Aspects of Design Single-source code Fortran90 modules, dynamic memory, structures, recursion Hierarchical software architecture Multi-level parallelism CASE: Registry Package-neutral APIs I/O, data formats Communication IKJ Storage Order WRF Software

  4. Software Architecture Registry Driver Driver Layer Config Inquiry DM comm I/O API Package Independent Solve Mediation Layer OMP Config Module WRF Tile-callable Subroutines Data formats, Parallel I/O Package Dependent Message Passing Threads Model Layer External Packages

  5. Single version of code for efficient execution on: Distributed-memory Shared-memory Clusters of SMPs Vector and microprocessors WRF Multi-Layer Domain Decomposition Logical domain 1 Patch, divided into multiple tiles Model domains are decomposed for parallelism on two-levels • Patch: section of model domain allocated to a distributed memory node • Tile: section of a patch allocated to a shared-memory processor within a node; this is also the scope of a model layer subroutine. • Distributed memory parallelism is over patches; shared memory parallelism is over tiles within patches Inter-processor communication

  6. Parallelism and you • Some code you add to WRF won’t entail any extra effort on your part other than conforming to the WRF model layer interface specification • Exceptions • Horizontal interpolations involving neighboring cells • Upper radiative boundary conditions • Dynamics and diffusion • Fully 3D packages (that is, not “column physics”) • Tell tale sign: if you have i+integer or i+integer array indices in right-hand side expressions in your code

  7. Example ! ! 0 pt 4 pt 8 pt 24 pt ! ! * * * * * ! * * * * * * * * * ! + * + * * + * * * + * * ! * * * * * * * * * ! * * * * * !msft x !rr x ! (dyn_eh/module_diffusion.F ) SUBROUTINE horizontal_diffusion_s (tendency, rr, var, . . . . . . DO j = jts,jte DO k = kts,ktf DO i = its,ite mrdx=msft(i,j)*rdx mrdy=msft(i,j)*rdy tendency(i,k,j)=tendency(i,k,j)- & (mrdx*0.5*((rr(i+1,k,j)+rr(i,k,j))*H1(i+1,k,j)- & (rr(i-1,k,j)+rr(i,k,j))*H1(i ,k,j))+ & mrdy*0.5*((rr(i,k,j+1)+rr(i,k,j))*H2(i,k,j+1)- & (rr(i,k,j-1)+rr(i,k,j))*H2(i,k,j ))- & msft(i,j)*(H1avg(i,k+1,j)-H1avg(i,k,j)+ & H2avg(i,k+1,j)-H2avg(i,k,j) & )/dzetaw(k) & ) ENDDO ENDDO ENDDO . . .

  8. Halo updates Distributed Memory Communications jme patch patch jms ime ims ims ime memory on one processor memory on neighboring processor

  9. Halo updates Distributed Memory Communications Halo update of second time level of rr may be defined in Registry as: halo HALO_DIFF dyn_em 4:rr_2 Halo update is invoked in the code (in the solve routine) as: #include “HALO_DIFF.incl” prior to the call to the diffusion routine * * + * * * memory on one processor memory on neighboring processor

  10. Halo updates Periodic boundary updates Period update of variable t may be defined in Registry as: period PBDY dyn_em 3:t Period update is invoked in the code as: #include “PBDY.incl” Distributed Memory Communications

  11. Halo updates Periodic boundary updates Parallel transposes Distributed Memory Communications all z on patch Transpose of u defined in Registry as: xpose TRANS_A dyn_em u,ux,uy Note separate definitions of u, ux, and uy: state real u ikj … state real ux ikjx … state real uy ikjy … Transpose from Z decomposition to X decomposition is invoked in the code as: #include “TRANS_A_z2x.incl” all y on patch all x on patch

  12. Software Architecture Registry Driver Driver Layer Config Inquiry DM comm I/O API Package Independent Solve Mediation Layer OMP Config Module WRF Tile-callable Subroutines Data formats, Parallel I/O Package Dependent Message Passing Threads Model Layer External Packages

  13. wrf (main/wrf.F) Driver Driver Layer integrate (frame/module_integrate.F) Config Inquiry DM comm I/O API Package Independent solve_interface (share/solve_interface.F) Solve Mediation Layer OMP solve_em (dyn_em/solve_em.F) Config Module WRF Tile-callable Subroutines Data formats, Parallel I/O Package Dependent Message Passing Threads Model Layer advance_uv (dyn_em/module_small_step_em.F) advance_uv (dyn_em/module_small_step_em.F) advance_uv (dyn_em/module_small_step_em.F) advance_uv (dyn_em/module_small_step_em.F) advance_uv (dyn_em/module_small_step_em.F) cumulus_driver (physics/cumulus_driver.F) External Packages Code Structure Registry

  14. Interface Mediation layer <> Model Layer All state arrays passed through argument list as simple (not derived) data types Domain, memory, and run dimensions passed unambiguously in three physical dimensions Restrictions on model layer subroutines No I/O, communication, no stops or aborts (use wrf_error_fatal in frame/module_wrf_error.F) No common/module storage of decomposed data (exception allowed for set-once/read-only tables) Spatial scope of a Model Layer call is one “tile” Temporal scope of a call is limited by coherency Computation on halos is allowed and considered a model-layer concern Benefits Ensures that a sub-domain is tile-callable and therefore portable to any decomposition supported by the driver Provides a common interface that Allows use of other conforming driver layers (this positions the code for new developments such as ESMF) WRF driver layer can (and is) being used to parallelize other models SUBROUTINE solve_xxx ( . . . !$OMP DO PARALLEL DO ij = 1, numtiles its = i_start(ij) ; ite = i_end(ij) jts = j_start(ij) ; jte = j_end(ij) CALL model_subroutine( arg1, arg2, . . . ids , ide , jds , jde , kds , kde , ims , ime , jms , jme , kms , kme , its , ite , jts , jte , kts , kte ) END DO . . . END SUBROUTINE template for model layer subroutine SUBROUTINE model_subroutine ( & arg1, arg2, arg3, … , argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims IMPLICIT NONE ! Define Arguments (S and I1) data REAL, DIMENSION (ims:ime,kms:kme,jms:jme) :: arg1, . . . REAL, DIMENSION (ims:ime,jms:jme) :: arg7, . . . . . . ! Define Local Data (I2) REAL, DIMENSION (its:ite,kts:kte,jts:jte) :: loc1, . . . . . . ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its,ids), MIN(ite,ide) loc(i,k,j) = arg1(i,k,j) + … END DO END DO END DO Driver DM comm Config I/O API Solve Inquiry OMP Data formats, Config WRF Tile - callable Threads Message Parallel I/O Passing Module Subroutines WRF Model Layer Interface

  15. template for model layer subroutine SUBROUTINE model ( & arg1, arg2, arg3, … , argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims IMPLICIT NONE ! Define Arguments (S and I1) data REAL, DIMENSION (ims:ime,kms:kme,jms:jme) :: arg1, . . . REAL, DIMENSION (ims:ime,jms:jme) :: arg7, . . . . . . ! Define Local Data (I2) REAL, DIMENSION (its:ite,kts:kte,jts:jte) :: loc1, . . . . . . ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its,ids), MIN(ite,ide) loc(i,k,j) = arg1(i,k,j) + … END DO END DO END DO • Domain dimensions • Size of logical domain • Used for bdy tests, etc.

  16. template for model layer subroutine SUBROUTINE model ( & arg1, arg2, arg3, … , argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims IMPLICIT NONE ! Define Arguments (S and I1) data REAL, DIMENSION (ims:ime,kms:kme,jms:jme) :: arg1, . . . REAL, DIMENSION (ims:ime,jms:jme) :: arg7, . . . . . . ! Define Local Data (I2) REAL, DIMENSION (its:ite,kts:kte,jts:jte) :: loc1, . . . . . . ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its,ids), MIN(ite,ide) loc(i,k,j) = arg1(i,k,j) + … END DO END DO END DO • Domain dimensions • Size of logical domain • Used for bdy tests, etc. • Memory dimensions • Used to dimension dummy arguments • Do not use for local arrays

  17. template for model layer subroutine SUBROUTINE model ( & arg1, arg2, arg3, … , argn, & ids, ide, jds, jde, kds, kde, & ! Domain dims ims, ime, jms, jme, kms, kme, & ! Memory dims its, ite, jts, jte, kts, kte ) ! Tile dims IMPLICIT NONE ! Define Arguments (S and I1) data REAL, DIMENSION (ims:ime,kms:kme,jms:jme) :: arg1, . . . REAL, DIMENSION (ims:ime,jms:jme) :: arg7, . . . . . . ! Define Local Data (I2) REAL, DIMENSION (its:ite,kts:kte,jts:jte) :: loc1, . . . . . . ! Executable code; loops run over tile ! dimensions DO j = jts, jte DO k = kts, kte DO i = MAX(its,ids), MIN(ite,ide) loc(i,k,j) = arg1(i,k,j) + … END DO END DO END DO • Domain dimensions • Size of logical domain • Used for bdy tests, etc. • Memory dimensions • Used to dimension dummy arguments • Do not use for local arrays • Tile dimensions • Local loop ranges • Local array dimensions

  18. Registry Registry Driver Driver Driver Layer Driver Layer Config Inquiry Config Inquiry DM comm DM comm I/O API I/O API Package Independent Package Independent Solve Solve Mediation Layer Mediation Layer OMP OMP Config Module Config Module WRF Tile-callable Subroutines WRF Tile-callable Subroutines Data formats, Parallel I/O Data formats, Parallel I/O Package Dependent Package Dependent Message Passing Message Passing Threads Threads Model Layer Model Layer External Packages External Packages Registry

  19. CASE-like “active data-dictionary” for managing WRF data structures Database describing attributes of model state, intermediate, and configuration data Dimensionality, number of time levels, staggering Association with physics I/O classification (history, initial, restart, boundary) Communication points and patterns Configuration lists (e.g. namelists) Program for auto-generating sections of WRF from database: 570 Registry entries  30-thousand lines of automatically generated WRF code Allocation statements for state data, I1 data Argument lists for driver layer/mediation layer interfaces Interprocessor communications: Halo and periodic boundary updates, transposes Code for defining and managing run-time configuration information Code for forcing, feedback and interpolation of nest data Automates time consuming, repetitive, error-prone programming Insulates programmers and code from package dependencies Allow rapid development Documents the data WRF Registry # Registry file # # table entries are of the form #<Table> <Type> <Sym> <Dims> <Use> <NumTLev> <Stagger> <IO> # state real ru ikj dyn-rk 2 X irh state real u ikj dyn-rk 2 X irh state real ru_m ikj dyn-rk 1 X i1 real ru_tend ikj dyn-rk 1 X . . . ## communications halo HALO_RK_A 24:u_2,v_2,w_2,t_2,tp_2,rw_2,rom_2;4:pp,pip halo HALO_RK_B 4:rtp_2 halo HALO_RK_C 4:ru_2,rv_2,du,dv . . .

  20. registry program: tools/registry Registry/Registry %compile wrf inc/*.incl Fortran90 WRF source */*.F wrf.exe Registry Mechanics

  21. Registry Data Base • Currently implemented as a text file: Registry/Registry • Types of entry: • State – Describes state variables and arrays in the domain structure • Dimspec– Describes dimensions that are used to define arrays in the model • I1 – Describes local variables and arrays in solve • Typedef– Describes derived types that are subtypes of the domain structure • Rconfig – Describes a configuration (e.g. namelist) variable or array • Package – Describes attributes of a package (e.g. physics) • Halo– Describes halo update interprocessor communications • Period– Describes communications for periodic boundary updates • Xpose– Describes communications for parallel matrix transposes

  22. State entry • Elements • Entry: The keyword “state” • Type: The type of the state variable or array (real, double, integer, logical, character, or derived) • Sym: The symbolic name of the variable or array • Dims: A string denoting the dimensionality of the array or a hyphen (-) • Use: A string denoting association with a solver or 4D scalar array, or a hyphen • NumTLev: An integer indicating the number of time levels (for arrays) or hypen (for variables) • Stagger: String indicating staggered dimensions of variable (X, Y, Z, or hyphen) • IO: String indicating whether and how the variable is subject to I/O • DName: Metadata name for the variable • Units: Metadata units of the variable • Descrip: Metadata description of the variable • Example # Type Sym Dims Use Tlev Stag IO Dname Descrip # definition of a 3D, two-time level, staggered state array state real ru ikj dyn_eh 2 X irh "RHO_U" "X WIND COMPONENT“ # definition of fields in 4D scalar array “moist” state real qv ikjft moist 2 - irh "QVAPOR" "Water vapor mix ratio“ state real qc ikjft moist 2 - irh "QCLOUD" "Cloud water mixing ratio"

  23. Dimspec entry • Elements • Entry: The keyword “dimspec” • DimName: The name of the dimension (single character) • Order: The order of the dimension in the WRF framework (1, 2, 3, or ‘-‘) • HowDefined: specification of how the range of the dimension is defined • CoordAxis: which axis the dimension corresponds to, if any (X, Y, Z, or C) • DatName: metadata name of dimension • Example #<Table> <Dim> <Order> <How defined> <Coord-axis> <DatName> dimspec i 1 standard_domain x west_east dimspec j 3 standard_domain y south_north dimspec k 2 standard_domain z bottom_top dimspec l 2 namelist=num_soil_layers z soil_layers

  24. Rconfig entry • Elements • Entry: the keyword “rconfig” • Type: the type of the namelist variable (integer, real, logical – no strings yet) • Sym: the name of the namelist variable or array • Howset: indicates how the variable is set: e.g. namelist or derived, and if namelist, which block of the namelist it is set in • Nentries: specifies the dimensionality of the namelist variable or array. If 1 (one) it is a variable and applies to all domains; otherwise specify max_domains (which is an integer parameter defined in module_driver_constants.F). • Default: the default value of the variable to be used if none is specified in the namelist; hyphen (-) for no default • Example # Type Sym How set Nentries Default rconfig integer dyn_opt namelist,namelist_01 1 1

  25. Package Entry • Elements • Entry: the keyword “package”, • Packagename: the name of the package: e.g. “kesslerscheme” • Associatedrconfigchoice: the name of a rconfig variable and the value of that variable that choses this package • Packagestatevars: unused at present; specify hyphen (-) • Associated 4D scalars: the names of 4D scalar arrays and the fields within those arrays this package uses • Example # specification of microphysics options package passiveqv mp_physics==0 - moist:qv package kesslerscheme mp_physics==1 - moist:qv,qc,qr package linscheme mp_physics==2 - moist:qv,qc,qr,qi,qs,qg package ncepcloud3 mp_physics==3 - moist:qv,qc,qr package ncepcloud5 mp_physics==4 - moist:qv,qc,qr,qi,qs # namelist entry that controls microphysics option rconfig integer mp_physics namelist,namelist_04 max_domains 0

  26. Comm entries: halo and period • Elements • Entry: keywords “halo” or “period” • Commname: name of comm operation • Description: defines the halo or period operation • For halo: npts:f1,f2,...[;npts:f1,f2,...]* • For period: width:f1,f2,...[;width:f1,f2,...]* • Example # first exchange in eh solver halo HALO_EH_A dyn_em 24:u_2,v_2,ru_1,ru_2,rv_1,rv_2,w_2,t_2;4:pp,pip # a periodic boundary update period PERIOD_EH_A dyn_em 2:u_1,u_2,ru_1,ru_2,v_1,v_2,rv_1,rv_2,rw_1,rw_2

  27. I/O Architecture Registry Driver Driver Layer Config Inquiry DM comm I/O API Package Independent Solve Mediation Layer OMP Config Module WRF Tile-callable Subroutines Data formats, Parallel I/O Package Dependent Message Passing Threads Model Layer External Packages

  28. I/O Architecture Registry Driver Driver Layer Config Inquiry DM comm I/O API Package Independent Solve Mediation Layer OMP Config Module WRF Tile-callable Subroutines Data formats, Parallel I/O Package Dependent Message Passing Threads Model Layer External Packages

  29. I/O Architecture • Requirements of I/O Infrastructure • Efficiency: key concern for operations • Flexibility: key concern in research • Both types of user-institution already heavily invested in I/O infrastructure • Operations: GRIB, BUFR • Research: NetCDF, HDF • “Portable I/O” – adaptable to range of uses, installations without affecting WRF and other programs that use the I/O infrastructure

  30. I/O Architecture • WRF I/O API • Package-independent interface to NetCDF, Fast-binary, HDF (planned) • Random access of fields by timestamp/name • Full transposition to arbitrary memory order • Built-in support for read/write of parallel file systems (planned) • Data-set centric, not file-centric (planned); Grid Computing • Additional WRF model functionality • Collection/distribution of decomposed data to serial datasets • Fast, asynchronous, “quilt-server” I/O from NCEP Eta model

  31. WRF I/O API • Key functions • Initialization and shutdown • Opening and closing datasets • Reading and writing data • Reading and writing metadata

  32. WRF I/O API 7.4.1 Initialization and shutdown SUBROUTINE ext_pkg_ioinit( Status ) INTEGER, INTENT(INOUT) :: Status Synopsis: Initialize the WRF I/O system. SUBROUTINE ext_pkg_ioexit( Status ) INTEGER, INTENT(INOUT) :: Status Synopsis: Shut down the WRF I/O system.

  33. WRF I/O API 7.4.2 Opening and closing datasets SUBROUTINE ext_pkg_open_for_read ( DatasetName , Comm , IOComm_io, & SysDepInfo , DataHandle , Status ) CHARACTER *(*), INTENT(IN) :: DatasetName INTEGER , INTENT(IN) :: Comm , IOComm CHARACTER *(*), INTENT(IN) :: SysDepInfo INTEGER , INTENT(OUT) :: DataHandle INTEGER , INTENT(OUT) :: Status Synopsis: Opens a WRF dataset for reading using the I/O package pkg. Arguments: DatasetName: The name of the dataset to be opened. Comm: Not used. IOComm: Not used. SysDepInfo: A string-valued variable that provides a means to pass additional control information to the I/O interface. The format of the string is a comma-separated list of variable=value pairs. Currently the only control variable now supported is DATASET, which may have the allowable values RESTART, HISTORY, BOUNDARY, or INITIAL (capitals required). For example, the WRF model will pass the string 'DATASET=INITIAL' to open an initial dataset. DataHandle: Returned to the calling program to serve as a handle to the Open dataset for subsequent I/O API operations.

  34. WRF I/O API SUBROUTINE ext_pkg_open_for_write_begin( DatasetName , Comm, IOComm, & SysDepInfo, DataHandle , Status ) CHARACTER *(*), INTENT(IN) :: DatasetName INTEGER , INTENT(IN) :: Comm, IOComm CHARACTER *(*), INTENT(IN) :: SysDepInfo INTEGER , INTENT(OUT) :: DataHandle INTEGER , INTENT(OUT) :: Status SUBROUTINE ext_pkg_open_for_write_commit( DataHandle , Status ) INTEGER , INTENT(IN ) :: DatasetHandle INTEGER , INTENT(OUT) :: Status Synopsis: Begin data definition phase for the WRF dataset DatasetName using the I/O package Pkg. Arguments: DatasetName: The name of the dataset to be opened. Comm: Not used. IOComm: Not used. SysDepInfo: Same as open for read. DataHandle: integer handle. Two-Stage Open For Write There is significant performance advantage for netCDF and presumably other self-describing data format packages in going through a data-definition phase prior to writing data through the package. Thus, the open-for-write is a two-stage operation. A WRF dataset is first opened for data-definition, at which point the model calls the interface (predominantly the ext_pkg_write_field routine) with a complete serious of writes such as would occur during the output of an actual frame of data; however, no data is actually written. This is strictly for the purpose of “training” the interface. Interface implementations that don’t need training can ignore this by simply opening the dataset on the open-begin call and setting an internal flag that disables writing in the write_field routines. ext_pkg_open_for_write_begin must be paired with an ext_pkg_open_for_write_commit .

  35. WRF I/O API SUBROUTINE ext_pkg_write_field ( DataHandle , DateStr , VarName , & Field , FieldType , Comm , IOComm & DomainDesc , MemoryOrder , Stagger , & DimNames , & DomainStart , DomainEnd , & MemoryStart , MemoryEnd , & PatchStart , PatchEnd , & Status ) INTEGER ,INTENT(IN) :: DataHandle CHARACTER*(*) ,INTENT(IN) :: DateStr CHARACTER*(*) ,INTENT(IN) :: VarName type ,INTENT(IN) :: Field(*) INTEGER ,INTENT(IN) :: FieldType INTEGER ,INTENT(INOUT) :: Comm INTEGER ,INTENT(INOUT) :: IOComm INTEGER ,INTENT(IN) :: DomainDesc CHARACTER*(*) ,INTENT(IN) :: MemoryOrder CHARACTER*(*) ,INTENT(IN) :: Stagger CHARACTER*(*) , DIMENSION (*) ,INTENT(IN) :: DimNames INTEGER ,DIMENSION(*) ,INTENT(IN) :: DomainStart, DomainEnd INTEGER ,DIMENSION(*) ,INTENT(IN) :: MemoryStart, MemoryEnd INTEGER ,DIMENSION(*) ,INTENT(IN) :: PatchStart, PatchEnd INTEGER ,INTENT(OUT) :: Status

  36. WRF I/O API SUBROUTINE ext_pkg_write_field ( DataHandle , DateStr , VarName , & Field , FieldType , Comm , IOComm & DomainDesc , MemoryOrder , Stagger , & DimNames , & DomainStart , DomainEnd , & MemoryStart , MemoryEnd , & PatchStart , PatchEnd , & Status ) INTEGER ,INTENT(IN) :: DataHandle CHARACTER*(*) ,INTENT(IN) :: DateStr CHARACTER*(*) ,INTENT(IN) :: VarName type ,INTENT(IN) :: Field(*) INTEGER ,INTENT(IN) :: FieldType INTEGER ,INTENT(INOUT) :: Comm INTEGER ,INTENT(INOUT) :: IOComm INTEGER ,INTENT(IN) :: DomainDesc CHARACTER*(*) ,INTENT(IN) :: MemoryOrder CHARACTER*(*) ,INTENT(IN) :: Stagger CHARACTER*(*) , DIMENSION (*) ,INTENT(IN) :: DimNames INTEGER ,DIMENSION(*) ,INTENT(IN) :: DomainStart, DomainEnd INTEGER ,DIMENSION(*) ,INTENT(IN) :: MemoryStart, MemoryEnd INTEGER ,DIMENSION(*) ,INTENT(IN) :: PatchStart, PatchEnd INTEGER ,INTENT(OUT) :: Status Data names and time stamps The variable Name provides the name of the variable being read/written; the DateStr character string is the index into the series of values of the variable/array in the dataset. In WRF, this is a 23 character date string of the form “0000-01-01_00:00:00.0000” and is treated as a temporal index. An external implementation of the WRF I/O API is required to randomly access a dataset based on the name and timestamp pair specified in the argument list. From the point of view of a package implementing the I/O API, the timestamp string has no meaning or relationship with any other timestamp other than lexicographic: two strings are either the same or they are different; there is no meaning associated with the timestamp string. The interface does not do time comparisons but it is aware of control breaks (change in the sequence of timestamps in the interface). The interface may also store a record of the sequence in which fields were written to a dataset and provide this information in a way that allows sequential traversal through the dataset but datasets are fundamentally random access by variable name and timestamp string. The interface may assume and expect that all the fields in a frame of data have the same date string and that on writing the calls to the interface to write them will have been called in consecutively.

  37. Full 2-D/3D Arrays 1-D (Column) Arrays Non-decomposed Boundary 3-D Boundary 2-D ‘XYZ’ ‘Z’ ‘C’ ‘XSZ’ (west) ‘XS’ (west) ‘XZY’ ‘XEZ’ (east) ‘XE’ (east) ‘ZXY’ 0-D (Scalars) ‘YSZ’ (south) ‘YS’ (south) ‘XY’ ‘0’ (zero) ‘YEZ’ (north) ‘YE’ (north) ‘YX’ WRF I/O API SUBROUTINE ext_pkg_write_field ( DataHandle , DateStr , VarName , & Field , FieldType , Comm , IOComm & DomainDesc , MemoryOrder , Stagger , & DimNames , & DomainStart , DomainEnd , & MemoryStart , MemoryEnd , & PatchStart , PatchEnd , & Status ) INTEGER ,INTENT(IN) :: DataHandle CHARACTER*(*) ,INTENT(IN) :: DateStr CHARACTER*(*) ,INTENT(IN) :: VarName type ,INTENT(IN) :: Field(*) INTEGER ,INTENT(IN) :: FieldType INTEGER ,INTENT(INOUT) :: Comm INTEGER ,INTENT(INOUT) :: IOComm INTEGER ,INTENT(IN) :: DomainDesc CHARACTER*(*) ,INTENT(IN) :: MemoryOrder CHARACTER*(*) ,INTENT(IN) :: Stagger CHARACTER*(*) , DIMENSION (*) ,INTENT(IN) :: DimNames INTEGER ,DIMENSION(*) ,INTENT(IN) :: DomainStart, DomainEnd INTEGER ,DIMENSION(*) ,INTENT(IN) :: MemoryStart, MemoryEnd INTEGER ,DIMENSION(*) ,INTENT(IN) :: PatchStart, PatchEnd INTEGER ,INTENT(OUT) :: Status

  38. WRF I/O API SUBROUTINE ext_pkg_write_field ( DataHandle , DateStr , VarName , & Field , FieldType , Comm , IOComm & DomainDesc , MemoryOrder , Stagger , & DimNames , & DomainStart , DomainEnd , & MemoryStart , MemoryEnd , & PatchStart , PatchEnd , & Status ) INTEGER ,INTENT(IN) :: DataHandle CHARACTER*(*) ,INTENT(IN) :: DateStr CHARACTER*(*) ,INTENT(IN) :: VarName type ,INTENT(IN) :: Field(*) INTEGER ,INTENT(IN) :: FieldType INTEGER ,INTENT(INOUT) :: Comm INTEGER ,INTENT(INOUT) :: IOComm INTEGER ,INTENT(IN) :: DomainDesc CHARACTER*(*) ,INTENT(IN) :: MemoryOrder CHARACTER*(*) ,INTENT(IN) :: Stagger CHARACTER*(*) , DIMENSION (*) ,INTENT(IN) :: DimNames INTEGER ,DIMENSION(*) ,INTENT(IN) :: DomainStart, DomainEnd INTEGER ,DIMENSION(*) ,INTENT(IN) :: MemoryStart, MemoryEnd INTEGER ,DIMENSION(*) ,INTENT(IN) :: PatchStart, PatchEnd INTEGER ,INTENT(OUT) :: Status Staggering Staggered means with respect to mass-point coordinates. For example, in WRF the Eulerian height solver uses an Arakawa C-grid so that the U-velocity is staggered in X, the V-velelocity is staggered in Y, and the vertical velocity are staggered in Z. The stagger string is specified to the read_field and write_field using a string consisting of the characters X, Y, and/or Z that specifies which dimensions the field is staggered in (or the empty string if the variable is not staggered in any dimension).

  39. WRF I/O API SUBROUTINE ext_pkg_write_field ( DataHandle , DateStr , VarName , & Field , FieldType , Comm , IOComm & DomainDesc , MemoryOrder , Stagger , & DimNames , & DomainStart , DomainEnd , & MemoryStart , MemoryEnd , & PatchStart , PatchEnd , & Status ) INTEGER ,INTENT(IN) :: DataHandle CHARACTER*(*) ,INTENT(IN) :: DateStr CHARACTER*(*) ,INTENT(IN) :: VarName type ,INTENT(IN) :: Field(*) INTEGER ,INTENT(IN) :: FieldType INTEGER ,INTENT(INOUT) :: Comm INTEGER ,INTENT(INOUT) :: IOComm INTEGER ,INTENT(IN) :: DomainDesc CHARACTER*(*) ,INTENT(IN) :: MemoryOrder CHARACTER*(*) ,INTENT(IN) :: Stagger CHARACTER*(*) , DIMENSION (*) ,INTENT(IN) :: DimNames INTEGER ,DIMENSION(*) ,INTENT(IN) :: DomainStart, DomainEnd INTEGER ,DIMENSION(*) ,INTENT(IN) :: MemoryStart, MemoryEnd INTEGER ,DIMENSION(*) ,INTENT(IN) :: PatchStart, PatchEnd INTEGER ,INTENT(OUT) :: Status Dimension Information Each of the six arguments show are integer arrays whose length is the number of dimensions of the data item being written (specified by the MemoryOrder string). DomainStart and Domain are the starts and ends of the undecomposed array in each dimension. MemoryStart and MemoryEnd are the starts and ends of each local array dimension as it is declared in the program. PatchStart and PatchEnd are the starts and ends of the local subdomain on this process.

  40. call ext_ncd_ioinit(Status) call ext_ncd_open_for_write_begin ( "WRF_DATA", 0, 0, "", dh1, Status ) start_memory(1) = ims ; end_memory(1) = ime start_memory(2) = kms ; end_memory(2) = kme start_memory(3) = jms ; end_memory(3) = jme start_domain(1) = 1 ; end_domain(1) = ide-1 start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde-1 call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","T",t,WRF_REAL,0,0,0, & "XZY"," ", & dimnames , & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !patch Status) start_domain(1) = 1 ; end_domain(1) = ide start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde-1 call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","U",u,WRF_REAL,0,0,0, & "XZY","X", & dimnames_x , & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !patch Status) start_domain(1) = 1 ; end_domain(1) = ide-1 start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","V",v,WRF_REAL,0,0,0, & "XZY","Y", & dimnames_y , & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !patch Status) call ext_ncd_open_for_write_commit ( dh1, Status ) ! file now open for writing WRF I/O API

  41. WRF I/O API ! file now open for writing start_domain(1) = 1 ; end_domain(1) = ide-1 start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde-1 call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","T",t,WRF_REAL,0,0,0, & "XZY"," ", & dimnames , & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !domain Status) start_domain(1) = 1 ; end_domain(1) = ide start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde-1 call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","U",u,WRF_REAL,0,0,0, & "XZY","X", & dimnames_x , & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !domain Status) start_domain(1) = 1 ; end_domain(1) = ide-1 start_domain(2) = 1 ; end_domain(2) = kde-1 start_domain(3) = 1 ; end_domain(3) = jde call ext_ncd_write_field(dh1,"2000-01-24_12:00:00","V",v,WRF_REAL,0,0,0, & "XZY","Y", & dimnames_y , & start_domain,end_domain, & !domain start_memory,end_memory, & !memory start_domain,end_domain, & !domain Status) call ext_ncd_ioclose(dh1, Status ) call ext_ncd_ioexit( Status ) . . .

  42. WRF I/O API Output from ncdump netcdf WRF_DATA { dimensions: Time = UNLIMITED ; // (1 currently) DateStrLen = 19 ; west_east = 199 ; south_north = 299 ; bottom_top = 34 ; west_east_stag = 200 ; south_north_stag = 300 ; variables: char Times(Time, DateStrLen) ; float T(Time, bottom_top, south_north, west_east) ; T:FieldType = 104 ; T:MemoryOrder = "XYZ" ; float U(Time, bottom_top, south_north, west_east_stag) ; U:FieldType = 104 ; U:MemoryOrder = "XYZ" ; float V(Time, bottom_top, south_north_stag, west_east) ; V:FieldType = 104 ; V:MemoryOrder = "XYZ" ; data: Times = "2000-01-24_12:00:00" ; . . .

  43. Additional Notes on WRF I/O • Calls for reading and writing time-dependent and time-independent meta data associated with datasets and individual variables • A sample reader program is distributed with WRF in external/io_netcdf/diffwrf.F90 • The WRF framework provides higher level I/O routines (see Sec. 7.2): • Higher level calls to read/write fields from an entire domain • Separate Initial, History, Restart and Boundary I/O streams; 4 auxiliary Initial and History streams • Separately run-time assignable I/O formats to I/O streams • Parallel/Quilted I/O • Integrated and controllable through the Registry; little to actually code

  44. Example: Adding a new core • Conceptual • WRF framework can slot in new dynamics as run-time selectable option • Changes to: • Mediation layer, model layer • Registry • Reuse: • Top-level driver layer • I/O infrastructure • Parallel infrastructure Registry

  45. Steps Develop new or convert existing code: Mediation layer routine: solve Model layer subroutines called by solver Add to WRF Add code to source tree Incorporate into build mechanism Registry entries: data, solver options, comms Some additional splicing Single processor testing Analyze data-dependencies, define and implement communication for parallelism Multi-processor testing Example: Adding a new core Registry Driver Driver DM comm DM comm Config Config I/O API I/O API Solve Solve Inquiry Inquiry OMP OMP Data formats, Data formats, Config Config WRF Tile WRF Tile - - callable callable Threads Threads Message Message Parallel I/O Parallel I/O Passing Passing Module Module Subroutines Subroutines

  46. PSEUDO CODE FOR NEW WRF SOLVER: Time loop outside of solver (part of WRF driver: integrate) DO 1 <- number of iterations subroutine solve_exp ( x_1 , x_2 ) for each i,j X1,i,j+1 X2,ij <= X1,i-1,j X1,i+1,j X1,i,j-1 for each i,j X1,ij <= X2,ij end subroutine solve_exp End time loop SOLVE_EXP

  47. !WRF:MEDIATION_LAYER:SOLVER SUBROUTINE solve_exp ( grid , & ! #include "exp_dummy_args.inc" ! ) CALL set_tiles ( grid , . . . ) #ifdef DM_PARALLEL # include "HALO_EXP_A.inc" #endif !$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1 , grid%num_tiles CALL comp_1_into_2 ( x_1, x_2, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO !$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1 , grid%num_tiles CALL copy_2_into_1 ( x_2, x_1, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO SOLVE_EXP

  48. !WRF:MEDIATION_LAYER:SOLVER SUBROUTINE solve_exp ( grid , & ! #include "exp_dummy_args.inc" ! ) CALL set_tiles ( grid , . . . ) #ifdef DM_PARALLEL # include "HALO_EXP_A.inc" #endif !$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1 , grid%num_tiles CALL comp_1_into_2 ( x_1, x_2, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO !$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1 , grid%num_tiles CALL copy_2_into_1 ( x_2, x_1, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO SOLVE_EXP

  49. !WRF:MEDIATION_LAYER:SOLVER SUBROUTINE solve_exp ( grid , & ! #include "exp_dummy_args.inc" ! ) CALL set_tiles ( grid , . . . ) #ifdef DM_PARALLEL # include "HALO_EXP_A.inc" #endif !$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1 , grid%num_tiles CALL comp_1_into_2 ( x_1, x_2, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO !$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1 , grid%num_tiles CALL copy_2_into_1 ( x_2, x_1, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO SOLVE_EXP

  50. !WRF:MEDIATION_LAYER:SOLVER SUBROUTINE solve_exp ( grid , & ! #include "exp_dummy_args.inc" ! ) CALL set_tiles ( grid , . . . ) #ifdef DM_PARALLEL # include "HALO_EXP_A.inc" #endif !$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1 , grid%num_tiles CALL comp_1_into_2 ( x_1, x_2, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO !$OMP PARALLEL DO & !$OMP PRIVATE ( ij ) DO ij = 1 , grid%num_tiles CALL copy_2_into_1 ( x_2, x_1, & ids, ide, jds, jde, kds, kde, & ims, ime, jms, jme, kms, kme, & grid%i_start(ij), grid%i_end(ij), & grid%j_start(ij), grid%j_end(ij), & k_start, k_end ) END DO SOLVE_EXP

More Related