WHAT IS PARAMETERIZATION UNCERTAINTY AND HOW TO EVALUATE IT FOR ENSEMBLE PREDICTION?

1. WHAT IS PARAMETERIZATION UNCERTAINTY AND HOW TO EVALUATE IT FOR ENSEMBLE PREDICTION? Tomislava Vukicevic AOML/HRD AOML

2. Common acknoledgments • Parameterizations of sub-grid-scale processes in NWP models are uncertain • This uncertainty leads to forecast uncertainty or errors • Using ensemble forecast based on different realizations of a single of multiple parameterizations would reduce forecast errors • Because ensemble mean forecast is expected to be more accurate than anyone deterministic forecast with a single version of the uncertain parameterization ✔ ✔ AOML

3. Premises for accurate ensemble predictionregarding the model error from uncertain parameterizations • Forecast ensemble is representative of actual forecast errordistribution • for each forecast scenario • Ensemble mean isnot biased • guaranteed smaller errors than anyone deterministic forecast Satisfied if • Ensemble is formed by sampling distribution of actual parameterization errors • for each forecast scenario • Relationshipsarenearly linear • between the forecast quantities and what is varied about the parameterizations AOML

4. Why is the nonlinearity a problem? • Nonlinear mapping via NWP model from the parameterization to forecast quantities would result in a skewed and often multi-modal distribution of the forecast quantities • The ensemble mean would not represent the most likely solution given the observations, thus would be biased • This problem would persist even with a perfect sample of parameterization perturbations AOML

5. How to achieve optimal ensemble ? Representative of the actual errors and unbiased • Objectively estimate uncertainty of quantities that control the parameterization performance using observations • By means of an estimation technique that would produce ensemble consistent with posterior statistical distribution • Find a representation of the parameterization uncertainty that would effectively linearize the relationship with the forecast AOML

6. What is parameterization uncertainty?Two types of uncertainty could be considered Epistemic Free parameters in the parameterization are uncertain dx/dt = p1(a) +p2(b) ….. Values of (a, b, …) control parameterization performance Aleatory Representation of processes is uncertain dx/dt = αp1 +βp2 ….. Values of (α, β, …) control parameterization performance Either set of parameters could be estimated using the observations AOML

7. What observations should be used? • With high information content about the processes that are parameterized • Equivalent to the observations that would be used for model validation • Often different from the key forecast quantities For example, TC track and intensity cannot be used to estimate uncertainty of microphysics or PBL parameterization in a NWP model because they do not contain the information about the state and processes of the microphysics or PBL AOML

8. Demonstration of the approach Results from a sequence of 4 studies on quantifying the uncertainty of a microphysics parameterization using satellite and radar observationsPosselt and Vukicevic, 2010van Lier-Walqui et al,, 2012van Lier-Walqui et al., 2014Vukicevic, et al., 2014/15 AOML

9. In the studies a fully nonlinear and accurate, but computationally expensive, Markov Chain Monte Carlo (MCMC) technique was used to evaluate both types of uncertainty • Bayesian estimate based on a very large sample • Includes all joint control parameter values that have non-zero probability of producing the forecast that falls within the observation error variance • Results are expressed in terms of multi-dimensional PDF of parameter values • The final study tested utility of EnKF technique that could be applied in practice with full blown NWP model AOML

10. Microphysics parameterization model Cloud Droplets (g/kg) Rain (g/kg) • 1 moment bulk microphysics (Tao et al, 2003) • Follow column through convective system in a Lagrangian sense • Captures transition from convective to stratiform Pristine Ice (g/kg) Snow (g/kg) Graupel (g/kg) AOML

11. Simulated Observations • Radar • Columns of reflectivity • Quickbeamradar simulator (Haynes et al, 2007) • Correlated observation errors • Used to perform estimation • Satellite-based products • Precipitation rate (PR) • liquid water path (LWP) • Ice water path (IWP) • Outgoing long wave (OLW) radiation • Outgoing short wave (OSW) radiation • Used as independent observations to evaluate ensemble forecast AOML

12. Epistemic uncertainty : Estimation of joint probability for 10 microphysical parameters AOML

13. Physical parameter uncertainty 2D view of 10D PDF • Uncertainty of physical parameters is complex and • nonlinear • Multimodal • Highly correlated • Thus, one parameter cannot be changed without changing all accordingly to be consistent with the model formulation and observations • Common mistake made • for physics ensembles or parameterization tuning is not to respect this • condition AOML

14. Transformation into process uncertainty • Using the ensemble based on the parameter uncertainty, the corresponding process uncertainty is examined • Unimodal, nearly symmetrical response distributions for the processes • Suggests linearization of the parameterization uncertainty in terms of process uncertainty using multiplicative stochastic coefficients • dx/dt = αp1 +βp2 ….. AOML

15. Parameterized processes to which the stochastic coefficients were assigned AOML

16. Comparison of physical and process parameter uncertainty estimates Physical parameters Process coefficients Melting of snow Melting of graupel Cloud ice accr of rain Graupel accr of rain Rain accr of cloud water Rain accr of snow Deposition of snow Depos/riming snow Snow accr of rain Graupel accr of rain Snow accr of cloud Graupel accr of cloud • The process coefficient PDF is much simpler, with weak or no correlations • Some processes show very little impact AOML

17. Is the process uncertainty linear?Test of Gaussianity using ‘Mahalanobis distance’ Deviations from Gaussianity (dashed line) are deviations from linearity Nearly linear The estimate of stochastic process coefficients corresponds to Log-Normal analytical distribution AOML

18. Impact on ensemble forecast Verification using independent observations Using the process uncertainty representation leads to unbiased ensemble forecast with small variance The parameter uncertainty leads to biased ensemble forecast and larger variance due to nonlinearity Aleatory Epistemic PrecRate LWP IWP OLW OSW • Results suggest that the parameterization uncertainty for the ensemble prediction may be well represented in terms of stochastic variations of the process tendencies • Consistent with demonstrated positive impacts of using a stochastic tendency term to represent model errors in global ensemble forecasting AOML

19. Summary regarding the representation of parameterization uncertainty • Physical parameter uncertainty is highly nonlinear • Non-Gaussian, mulitimodal distribution • Leads to biased ensemble forecast • Cannot be well estimated using practical ensemble data assimilation techniques • Using the stochastic process coefficients renders nearly linear estimation problem • Log-Normal uncertainty distribution • Supports unbiased and more certain ensemble forecast • May be estimated using practical ensemble DA techniques AOML

20. Toward practical application for a full blown NWP model • Ensemble Transform Kalman Filter (ETKF) technique was applied to both the physical and process parameter uncertainty estimation problems and compared to MCMC results • Gaussian and Log-Normal formulations of the ETKF algorithm were tested AOML

21. Comparison of MCMC and EnKF solutions for the process coefficients MCMC EnKF • Standard EnKFproduces similar result to MCMC but with less accuracy and precision AOML

22. Improvement from using Log-Normal EnKF • Using Log-Normal EnKF including Log-Normal prior produces more accurate posterior ensemble of coefficients • Using the Log-Normal prior makes most of the improvement AOML

23. Ensemble forecast verification • Comparison of forecast ensembles based on uncertainty estimates from MCMC and 2 flavors of EnKF solutions for both types of uncertainty Best from MCMC process uncertainty ensemble Worst from EnKF physical parameter uncertainty ensemble AOML

24. Ensemble generation using EnKF MCMC physical params MCMC process params EnKF physical params EnKF process params EnKF process params Log-Normal Prior PrecRate OSW • EnKF for physicals parameters performs worst due to the impact of nonlinearity, as expected (dark green curves): large bias and variance • Standard EnKF for process coefficients produces better ensemble but still inaccurate (orange curves) • Log-Normal EnKF with Log-Normal prior for the process coefficients produces the best ensemble (magenta curves): small bias and less variance AOML

25. Summary • Parameterization uncertainty could be estimated using observations that have high information content about the parameterized processes • Field program observations could be highly valuable for this purpose • The uncertainty of microphysics parameterization in terms of physical parameters is not well suited for the ensemble prediction because of the nonlinear relationships with the observations and forecast quantities • Representing the uncertainty in terms of stochastic process coefficients leads to nearly linear estimation problem • Supports nearly unbiased forecast ensemble • Could be easily implemented in ‘physics’ ensembles for any NWP model AOML

26. For practical application with NWP models • Ensemble Kalman Filter data assimilation could be used to estimate the stochastic process coefficients • Log-Normal EnKF formulation is needed for this purpose because the coefficients are positive-definite quantities • Specific formulation of the stochastic process coefficients must be investigated for each parameterization • The parameterization uncertainty could be performed retrospectively using special field program or other relevant observations OR in on-line mode during data assimilation cycling for initial condition analysis AOML

27. Example of parameterization uncertainty estimation with an idealized vortex model Rios-Berrios et al, 2014, MWR NHC

28. Model : ASPECH (Axisymmetric Simplified PseudoadiabaticEnthropy Conserving Hurricane model; Tang and Emanuel, 2012) • Uncertainty was estimated for latent heat of vaporization (Lh) and enthalpy exchange coefficient (Ce) using 2 types of observations • Tangential wind maximum • 2D kinematic field within vortex core • Estimation was performed using an ensemble inverse modeling technique similar to MCMC NHC

29. Dependence of uncertainty estimation on information content of observations Kinematic field observations Maximum wind observation Reference truth Ce Lh Lh • Relationship of the parameters to the maximum wind is highly nonlinear, leading to poorly constrained estimation problem • 2D kinematic field observations have linear relationship to the parameters and can constrain the uncertainty estimation NHC

30. Impact of nonlinearity on the forecast ensemble mean • Using the parameter perturbations not constrained with the kinematic field observations produces biased ensemble forecast due to the nonlinear relationship NHC

31. Summary • PBL parameter optimal values and uncertainty could be estimated using estimation with observations • Maximum wind observations cannot constrain well the parameters and their uncertainty because of the nonlinear relationship • Kinematic field observations such as produced from Doppler wind analysis and similar could be used to estimate the optimal parameter values and uncertainty for the ensemble forecasts • Parameter and ensemble optimization done in this way would also lead to optimal maximum wind forecast NHC

32. NHC

33. Practical challenges and recommendations • Estimating the optimal physical parameters within the parameterization is typically associated with highly nonlinear relationship with the forecast and most observations • Currently used practical data assimilation techniques are not well suited for that problem • Problem could be re-defined to estimation of the aleatory uncertainty : the uncertainty of parameterized process contributions • The aleatory uncertainty may be associated with less nonlinearity, allowing for the use of practical techniques • Careful exploration of the aleatory uncertainty is required for each parameterization using the nonlinear techniques for understanding the problem • Estimation should involve observations that are relevant to the parameterized process NHC