The Joint Effort for Data assimilation Integration (JEDI)

1. The Joint Effort for Data assimilation Integration (JEDI) • Model interfacing • Joint Center for Satellite Data Assimilation (JCSDA) • JEDI Academy - 10-13 June 2019

2. Outline Introduction Model space classes GetValues Building an application C++/Fortran binding

3. Introduction OOPS provides the algorithms that combine generic building blocks into applications such as variational assimilation, forecast, EnKF, FSOI etc. OOPS (by design) knows nothing about the actual implementation of the building blocks and carries no information about the underlying data. The classes that need to be implemented for a specific model are called interface classes. Often models are written in Fortran so a mixed language approach is required and a binding between the languages is implemented. Once the interface to a specific model is ready it can be used to create applications by passing traits and information about factories.

4. Models being interfaced to JEDI

5. From interface to implementation All the model (and UFO) classes follow basically the same file structure for the mixed C++/Fortran languages: StateFV3JEDI.h fv3jedi_state_interface_mod.F90 State.h (OOPS) StateFV3JEDI.cc fv3jedi_state_mod.f90 StateFV3JEDIFortran.h For a Fortran based model almost all the work and memory is here.

6. Model space classes

7. Model space interface classes Geometry State Increment InterpolatorTraj Model LinearModel ErrorCovariance Localization ModelAuxControl ModelAuxCovariance ModelAuxIncrement VariableChange LinearVariableChange Geometry, State, Increment, interpolation Models Background error covariance Model bias correction or parameter estimation (empty for now) Nonlinear and linear change of variables

8. Building the FV3-JEDI building blocks Model GEOS GEOS FV3-JEDI-LM NEMSfv3gfs LM NEMS GetValues ioda ufo State Geometry LinearModel FV3-JEDI-LM Increment ErrorCovariance/ Localization saber VarChange/ LinVarChange Poisson solver

9. Geometry Class: OOPS vs. FV3-JEDI OOPS FV3-JEDI

10. Geometry: Fortran interfaces

11. Geometry Class: Fortran Type

12. Geometry Class: Fortran Methods Call to dynamical core

13. State and Increment: Methods Increment State random self_add (+=) self_schur self_sub (-=) self_mul (*=) axpy_inc axpy_state dot_prod diff_states getValues_tl getValues_ad ug_coord increment_to_ug ug_to_increment dirac create delete zeros copy read write gpnorm rms change_res axpy add_increment getValues analytic_ic

14. Field Class The concept of fields is introduced in order to limit duplicate code across sate and increment. Some model interfaces implement fields at the C++ level, e.g. qg, l95, soca. Some do so at the Fortran level, e.g. fv3-jedi, mpas, lfric.

15. State/Increment variables

16. State/Increment constructor Incoming vars are decided by the user at run time. Variables are pre-programmed but not hardwired

17. State/Increment method Optionally check same list of fields in self and rhs Loop through all allocated fields. Not dependent on variables chosen.

18. Model class Factory name

19. Model class GEOS NEMSfv3gfs Dynamical core only Pseudo model

20. Data flow STATE MODEL MODEL STEP (EXTERNAL) MODEL STATE TRAJ & PPs Can be pointer or copy. Usually a copy to account for differences in precision

21. LinearModel class CREATE Per outer loop FINALIZE_AD INIT_TL STEP_TL STEP_AD Every call FINALIZE_TL INIT_AD DELETE

22. Variable changes Incremental hybrid-4DVar involves a number of linear and nonlinear variable transforms:

23. Variable changes Sets of variables: Background: the variables which you end up with an analysis of. Typically chosen to interact well with the forecast model being restarted. Control increment: the variables of , chosen based on various considerations. Model: the variables that the model and linear model need, e.g. staggered winds. B matrix variables: the variables used in the B matrix, e.g. unbalanced stream function and velocity potential.

24. VarChaC2MFV3JEDI Increment containing control variables comes in, increment with model variables goes out. The base class handles the allocation and deallocation either side.

25. Get Values

26. OOPS – UFO – IODA – MODEL: the interface advantage • JEDI/UFO introduces standard interfaces between the model and observation worlds. • Observation operators are independent of the model, easy sharing, exchange and comparison. Observation Locations Observation space MODEL IODA getValues getValuesTL getValuesAD Variables Observation operator OOPS Observation vector GeoVals: State values at locations UFO

27. GetValues • Get Values is the routine by which the model provides the GeoVaLs to the generic part of the observation operator (UFO). • It can be thought of as the grid/model dependent part of the observation operator – providing the variables called for by UFO at the locations of the observations. • Currently it involves some variable changes, though that will be moved. It also contains interpolation from the model native grid to observation locations. • The observations operators to be instantiated are chosen through the YAML, if getValues does not recognize the variable being asked for the system should abort.

28. getValues: Fortran interfaces • Geometry • State/increment • Locations • Variables • Geo Values • Trajectory for TL/AD

29. InterpolatorTraj Trajectory needed for variable change Interpolation object

30. GetValues: algorithm Compute weights for interpolation Loop over UFO variables Select case on variable Convert variable and prepare interpolation Loop over levels Interpolate to locations end (levels) End (variables)

31. getValues: prepare state/increment variable Loop over variables Flag on whether to interpolate Set number of levels for variable and interpolation flag. Transform the variable if need be Some variables use integration or aren’t float. ABORT

32. getValues: allocate GeoVaLs The model has to allocate the GeoVaLs since it knows the number of model levels.

33. getValues: BUMP interpolation weights Unstructured vector of model grid points.

34. getValues: BUMP interpolation apply Step 1: Convert to vector. Same structure as grid is organized when passed to BUMP setup. Call BUMP Fill up GeoVaLs with locations in current time bin.

35. getValues: BUMP adjoint interpolation 1. Set pointer 2. Apply interpolation weights 3. Convert back to increment.

36. Building an application

37. Building an application driver Geometry If an oops branch with a bug is merged and no one is there to compile it, does the bug really exist? State Increment OOPS fv3jedi_var.x Traits/ Factories Driver fv3jediVar.cc Model … oops on its own is just headers requiring a template to be applied to the interface classes and be passed via traits and factories. ufo/ioda

38. fv3jediVar.cc application driver Include the model traits Include the factories Include the main application Pass config (YAML) Initialization step (FMS etc) Instantiate factories Create application object Execute application

39. Run Subclass can perform any local initialization steps. E.g. initialize library required by FV3 Inheritance from the base class Run This can do generic initialization such as MPI init and prepare generic monitoring tools. In turn inherits eckit::Main. Execute runs the application and diagnostics.

40. Variational.h

41. Run execute Receives and application Calls application execute Checking of proper run Output some diagnostics

42. FV3-JEDI Traits OOPS level State.h interface FV3-JEDI Templates passed in through Traits. Basically just a list of implemented classes.

43. Factory instantiation Instantiate the change of variable designated by VarChaC2MFV3JEDI. In the YAML we need to call as “Control2Model” Factory: the class VarChaC2MFV3JEDI is then implemented as normal. YAML: choose the subclass and the variables to be allocated.

44. C++/Fortran binding

45. Binding: C++ side GeometryFV3JEDI.cc GeometryFV3JEDIFortran.h

46. Binding: Fortran side fv3jedi_geom_interface_mod.F90 Access to the object is through a linked list Integer Key comes in, pointer to an object gets passed.

47. LinkedList inclusion At the interface_modlevel the Linked List is created for the Fortran version of the object. linkedList_i.fcontains the list of objects and linkedList_c.fcontains the methods for manipulating and accessing the current object in the linked list.

48. linkedList_i.f Linked list node is where an object is actually stored in memory. It also contains a pointer to the next element. Class containing pointer to the head node. Methods for accessing that object.

49. linkedList_c.f: initilaize If linked list not initialized associate the head node and set the flags.

50. linkedList_c.f: add Key comes in from OOPS. Adding an object to the linked list so ‘up the counter’ and set the key. Then allocate the next element. This is the actual allocation of memory for the object. Associate a pointer to the next element in the linked list.