Statistical modelling for planning and policy applications

1. Statistical modelling for planning and policy applications Dr. Chris Dent Chancellor’s Fellow and Reader in Industrial Mathematics Turing Fellow at the Alan Turing Institute Thanks to Amy Wilson, Stan Zachary, Michael Goldstein – and too many others to name

2. Why am I talking about this? • It matters! • Variability/uncertainty of renewables • Uncertainty in planning background • Linking models to the world • I am not a statistician! • BA Maths, PhD Physics, MSc OR, CEng, FORS, SMIEEE • What I learned about modelling from Physics • Not much studied at INI – or anywhere! • Stats work mostly short term forecasting • Planning work mostly optimisation not uncertainty specification • Working out what models tell us about world is unfashionable • Statistical thinking in resource adequacy: Zachary, Wilson, Dent?, Tindemans?, Gorinevsky, Kujala, Ekisheva, Astrape?

3. Contents • Resource adequacy • Inference from data • Large computer models • Capital planning • Policy – what if background very complex? • Where next? Scoping work: • Centre for Digital Built Britain network on planning under uncertainty • Turing Institute project on use of models in organisations • Earlier energy policy report

4. Resource adequacy Inference from data

5. Risk of absolute supply shortages

6. GB CA study probability model • Evolution of margin through peak season • available (conventional, VG) – interconnection, storage? • demand, time resolution • ranges over future season under study • Risk indices • Expected duration of shortfall in a future season (19-20, or 23-24) • Expected energy (volume of demand) not supplied • Probability distribution of duration of shortfall • Probability distribution of energy not supplied • Monetary quantification of cost of shortfalls • This is basically applied stats • Much open space for research

7. Lots of numbers… but how much information? Lots of numbers… but how much information

8. Source: Wilson/Zachary How much (relevant) data?

9. Limited relevant data • Few data points at times of high demand/low wind • 7 years of data used here • Approaches based on using distribution of • Hindcast – use historic empirical distribution (EVT variants) • General independence? • Use longer meteorological time series?

10. Source: National Grid Interest in risk level conditional on severity of winter High level decision makers and analysts Meaningful to make point estimate of ‘long-run’ LOLE? How to interpret list of examples of historic years? Mapping historic years’ demands to common future scenario Relevance of frequentist interpretation of probability? Limited relevant years

11. Calculating the right thing? Lots of numbers… but how much information

12. Indices and decision analysis • Expected value indices used in most industrial studies • Except Belgium! See National Grid report • http://sites.ieee.org/pes-rrpasc/files/2016/08/Daniel-Burke-National-Grid-GB-Security-of-Supply-International-Study-on-Standards-and-Implementation.pdf • But they are not everything… • ‘How bad can things credibly get?’ • No information on variability about mean • Generally expected money does not represent real decision makers • Time series modelling required! • Example • Fix LOLE • Vary amount of wind • Variability of EU

13. Validate models!!! Lots of numbers… but how much information

14. Distributions should be the same Source: Kevin Carden / Astrape

15. Extremes matter! Lots of numbers… but how much information

16. 1 system, time-collapsed 1 system, time series 2 systems Examples of EVT applications

17. Large computer models Capital planning Policy – what if background very complex?

18. Parametric uncertainty (each model requires a, typically high dimensional, parametric specification) Condition uncertainty (uncertainty as to boundary conditions, initial conditions, and forcing functions) Functional uncertainty (model evaluations take a long time, so the function is unknown almost everywhere) Stochastic uncertainty (either the model is stochastic, or it should be) Solution uncertainty (as the system equations can only be solved to some necessary level of approximation) Structural uncertainty (the model only approximates the physical system) Measurement uncertainty (as the model is calibrated against system data all of which is measured with error) Multi-model uncertainty (usually we have not one but many models related to the physical system) Decision uncertainty (to use the model to influence real world outcomes, we need to relate things in the world that we can influence to inputs to the simulator and through outputs to actual impacts. These links are uncertain.) Uncertainty about what is meant by uncertainty and probability Sources of uncertainty (MG)

19. Toy example by Amy Wilson – model Emulator quantifies uncertainty in value of function for all Interested in e.g. how propagates to uncertainty in Uncertainty in combines with that in for given Avoid dependence on choice of evaluations Framework to consider full range of uncertainties Statistical emulators

20. Great interest in contribution of interconnection to GB (Other systems will have their equivalent issues) Two linked issues Uncertainty in margin in Europe greater than link capacity ENTSO-e models complex and high dimensional How to model this? And multivariate time series with focus on extremes! Interconnectors and SoS

21. DEFRA – the Horrendogram

22. Capital planning optimisation Lots of numbers… but how much information

23. Antony Lawson Transmission capacity planning Meng Xu Calibration of generation investment projection model How to do optimization? Examples from power systems

24. Right for the right reason? Lots of numbers… but how much information

25. Distribution of event sizes https://doi.org/10.1109/PES.2008.4596715

26. Background – getting away from specific methodology for UQ etc There should be a clear statement of what the study is claiming to say about the real world… … along with a logical argument to back this up What proportion of studies satisfy (1) and (2)? Key requirements of an applied modelling study

27. Scoping activity

28. Attendees from government, industry, academia Report available (email me!) Key outputs – clear demand for R+D Management of uncertainty in energy systems modelling Communication of modelling results (inc uncertainty) beyond technical modellers The role of modelling within policy processes, approaches to quality assurance Not a shopping list for specific technical work Alan Turing Institute project ‘Managing uncertainty in government modelling’ Energy policy scoping workshop

29. All the usual things – which do matter Uncertainty, communication, organisational matters More intriguing outcomes Enabling work on state of knowledge/practice More nuanced understanding of failure/success Communication is a two way process Resource allocated to analysis for strategic planning Scenarios, de-risking contracts, engineering standards Data availability – enabling activity Design funding calls to enable interdisciplinarity Newton Gateway: “Evidence based decisions” Similar issues, e.g. V complex backgrounds Part of RC research/project agenda CDBB, Turing and Newton

30. References • Wilson/Zachary on wind/demand: • http://sites.ieee.org/pes-rrpasc/working-groups/wg-on-lole-best-practices/ • Zachary on decision analysis, Wilson/Goldstein on UQ • http://icms.org.uk/workshops/energytutorialday • Lawson/Dent/Goldstein on transmission planning (multistage to follow) • http://dx.doi.org/10.1016/j.segan.2016.05.003 • Xu/Wilson/Dent on generation projection model calibration • http://dx.doi.org/10.1016/j.segan.2015.10.007 • Sheehy et al on distribution of outcome metrics • https://doi.org/10.1109/PMAPS.2016.7764199 • Dent on “What is a blackout?” • https://www.dur.ac.uk/dei/resources/briefings/blackouts/ • HM Treasury Aqua Book • https://www.gov.uk/government/publications/the-aqua-book-guidance-on-producing-quality-analysis-for-government • Energy policy report • https://www.research.ed.ac.uk/portal/en/publications/modelling-in-public-policy(354d2e01-4cbe-48d9-983c-0e4f193dbef2).html • Cambridge Energy Efficient Cities Initiative • https://www.eeci.cam.ac.uk/