Data Assimilation: from Meteorology to Space Weather and Applications

1. Data Assimilation: from Meteorology to Space Weather and Applications B. Khattatov, M. Murphy, M. Gnedin, J. Sheffel Fusion Numerics Inc; V. Yudin, National Center for Atmospheric Research Tim Fuller-Rowell, NOAA/SEC – CU/CIRES

2. Data Assimilation • Origins of data assimilation reside in estimation and optimal control theories, developed primarily as pure mathematical concepts • Arguably, it was the first computationally practical estimation method, Kalman filter, that allowed to put men on the moon • It became evident in weather forecasting that without a practical framework for combining imperfect models and observations (Kalman filter being one), forecasting becomes largely useless • Data assimilation methods developed in meteorology has been recently used in other areas of Earth sciences

3. Background • One such area is chemical data assimilation, that aims at studying composition, pollution, and emission sources • Arguing that analogies exist between neutral atmosphere composition and ionospheric ion/electron content, we proposed to the US Air Force to build a practical assimilative ionospheric model • Two other similar projects were under way in the US, one lead by Utah State University, the other by JPL/Caltech • Our effort has been funded by the US Air Force for a number of years by about $2MM and was relatively successful

4. Primary Components of Data Assimilation • A forward model • An observational operator: Hxy • Methods of combining a forward (forecasting) model state with observations: • Sequential (e.g. Kalman filter) • Variational (e.g., 4-D Var) • Ensemble • Methods for statistical validation of the system performance and tuning assimilation parameters (e.g., OmA & OmF analysis, c2 analysis)

5. Sequential Methods • Sequential methods aim at constraining the model state with most recent data, usually at each model time step • Once the data has been used, the “memory” of incorporated data is contained in the model error covariance matrix • Both constrained model state and the covariance matrix are evolved forward in time using the forecasting model for the state and some variant of the Kalman filter for the covariance matrix • At the next time step the process repeats

6. Kalman Filter In the assimilation mode every 10 minutes model electron densities are corrected with data from GPS reference network. Fusion Numerics Inc

7. Receiver Bias Estimation • A major problem with GPS measurements of ionospheric electron content is inter -frequency hardware biases. • These biases can be as large or larger than the measurements themselves, resulting, for example, in negative electron content. • The biases change in time in response to changing environmental conditions (T, humidity, etc) • Biases are currently estimated in the same (augmented) Kalman filter that is used to assimilate observations. Fusion Numerics Inc

8. Bias Estimation Augment control state with biases: Apply Kalman filter: Fusion Numerics Inc

9. Statistical Validation Techniques • Observations-minus-Analysis statistics (OmA, the smaller the better, must be Gaussian & unbiased) • Observations-minus-Forecast analysis (OmF, same, but will be larger than OmA) • c2 analysis – verifies that the implied and the true model error growth rates are approximately the same • Other application specific techniques….

10. The developed operational ionospheric modeling and assimilation system consists of : - core ionospheric forecasting model solves plasma mass, momentum and energy conservation equations on a global 3-D grid. - empirical models of electric/magnetic fields, and neutral composition and winds - data assimilation component corrects model simulation with GPS ground station data via a large-scale suboptimal Kalman filter. - data fetching components, housekeeping modules, time synchronization modules, visualization, etc

11. Forward Ionospheric Model • New code developed by Fusion Numerics • Object-oriented, written in C++ • ~100,000 lines of code, ~100 classes • Solves plasma momentum, mass, and energy conservation equations. • O+, H+, He+, NO+, N+, O2+, N2+ • Fixed 3-D grid in magnetic coordinates. • Can use real-time measured solar activity from NOAA Space Environment System. Fusion Numerics Inc

12. A Part of the Model Grid in Geographic Coordinates Fusion Numerics Inc

13. High-Resolution Local Grid

14. Examples of Model Fields There are approximately 150 different variables at each of ~1,000,000 model grid points. Fusion Numerics Inc

15. IGS Stations Used in the System

16. Total Electron Content Map

17. High-Resolution Local TEC

18. Validation • A subset of the available IGS stations is systematically withheld from the assimilation and used for validation. • The validation stations are rotated at random every 24 hours. • Slant TEC computed in the system is then compared with slant TEC from the validation stations. • Global average RMS error is ~1-5 TEC units

19. Daily Validation Statistics

20. Jicamarca Comparisons Fusion Numerics Inc

21. A GPS Positioning Engine Relying on the Assimilation System Ionospheric Specifications

22. Positioning Results With and Without WAAS

23. Forecasting Issues • In meteorology forecast is primarily an initial value problem • Only under quiet conditions ionosphere can be reasonably forecasted for hours or even days • Proper forecast must rely on driver forecast, which is notoriously hard (e.g., geomagnetically efficient CMEs)

24. Data Latencies Issues • In meteorology data can be incorporated in the models at several hour long intervals • Ionospheric conditions can change dramatically on scale of seconds • Most IGS data are delivered in near-real time, but are freely available to the public with latencies of days to hours • New protocol and software developed by BKG (NTRIP) makes it easier for operators of individual (e.g., non-IGS) GPS stations to make their data available in real time

25. Current State • The developed assimilation system is currently operational and is being used by the US Air Force Research Lab • It has also been licensed by the European Space Agency (ESA), used by a prominent Asian government for studies of a GPS augmentation system, used by several large GPS equipment manufactures for evaluating ionospheric impact on GPS, and attracted interest from government-associated organizations in China, Russia, South Korea, and India.

26. Thank you for your attention! • Always looking for new ways and new partners to use our system for research, educational and commercial purposes • Contact info: boris@fusionnumerics.com

27. GPS Receiver Bias Estimation Fusion Numerics Inc

28. http://www.fusionnumerics.com/ionosphere

29. In the ideal world, what would we need to build an ionospheric now- or fore-casting system? • A model of electrical and magnetic fields : 3-D Ex, Ey, Ez, Bx, By, Bz in the vicinity of the Earth • A model of photon energy spectrum ranging from gamma rays to infrared • A model of external 3-D particle precipitation flux • A coupled 3-D neutral wind model • A coupled 3-D neutral chemical composition model • A coupled 3-D ionospheric model accounting for composition, momentum, and energy of the ionosphere on scales of up to 100 meters to include Raleigh-Taylor instabilities • Possibly a wave propagation model (s) • Enough observations to constrain all these models • Data assimilation modules that can use these observations • Lots of talented people • Lots of money

30. Variational Techniques • Variational methods aim at finding a model state that provides a best fit between the model trajectory and all observations available over a certain time interval • As such, the model trajectory inside that time window tends to be smooth and exactly satisfying the model equations • Covariance matrices are not explicitly evolved and are usually computed by some other methods • The time window then “slides” forward in time

31. Sequential vs. Variational Techniques • For linear forward models and observations with Gaussian error statistics, properly implemented sequential and variational techniques are equivalent • For a particular non-linear systems and non-Gaussian statistics, one method or another might be more accurate • How to evaluate performance of a given assimilation strategy?