Power Grid Research at Pacific Northwest National Laboratory

1. Power Grid Research at Pacific Northwest National Laboratory Moe Khaleel Laboratory Fellow and Director Computational Sciences and Mathematics Division September 10, 2009 HPC User Forum

2. Overview • Introduction • Problems on the electrical grid • Need for HPC on the electrical grid • Operations • Integration of renewables • Cyber security • Need for HPC at vision level • State estimation • Multithreaded platforms for contingency analysis • Looking forward

3. Transform the way the U.S. generates, transmits, distributes and uses electricity • Current U.S. electricity infrastructure is inadequate for national energy priorities in the 21st century. Three main areas need to be addressed: • Capacity • Grid management (wide area, real-time) • Vulnerability, resiliency, reliability • The future state of the grid must be able to: • Substantially increase the integration of renewables • Reduce carbon emissions • Provide flexibility to enable electrification of transportation and reduce dependence on oil imports (substitute electricity for oil) • Respond to increased demand • Reality: The current grid infrastructure/operation is limited • Currently manage/engage grid at sub-optimal (service territory) level; can’t efficiently move electrons across large enough spaces • No ability to see performance across grid (lacks transparency) • Inability to integrate renewables; need storage, ability to offset, transmission across service territories where generated, load/dispatch renewables • System communication has been one-directional: supply to demand • Fragmented authority, control, market, function, regulation • Need to build new functionality and infrastructure into the grid • Future grid must be transformed while maintaining reliability and affordability (serving public good) 3

4. Transform the way the U.S. generates, transmits, distributes and uses electricity • Capacity: Transmission infrastructure cannot meet future load growth and large-scale renewable connectivity to grid • Utilities not incentivized to build physical infrastructure • Difficult to site & permit new transmission infrastructure • Renewable resource physically isolated from high grid transmission infrastructure • Grid Management: Unable to manage grid at national, interconnect scale • Large scale models that allow examination and optimization of future national grid do not exist • Integrated wide area models (variable renewable generation, energy storage, distributed generation, demand management) that describe real-time power flow and predict reliability do not exist • Ability to see and understand the grid at interconnections scale are limited; wide area grid performance is not accessible, transparent so can’t optimize supply and demand across limited service areas • Transparent real-time monitoring and operation currently not in place • Large-scale wind generation introduces significant variability • Large scale electric energy storage capability is limited – pumped hydro, flywheels, electro-chemical systems connected to and supporting the power grid • Vulnerability and Resiliency • Susceptible to cyber, other threats – can we prevent, respond to threats? • Resilient to catastrophic events – can we rapidly recover? 4

5. Power System Elements

6. Problems on the electrical grid • Inadequacy of current control center functions • Slow, not able to keep up with the change of the grid • Static, no dynamic information for real-time operations • Computational Issues with today’s grid operations • Real-time grid view: static • Not able to capture grid dynamics • Low computational efficiency: not keep up with system changing • No use of high-performance computing architectures

7. Problems on the electrical grid

8. The need for HPC on the electrical grid • Major HPC Architectures • Shared-memory architecture for extensive data sharing and un-uniform data access • Distributed-memory architecture for less data sharing and uniform data access • Hybrid architecture - re-configurable architecture: e.g. FPGA + shared-memory • Performance of Parallel Algorithms/Programs • Speedup/Scalability • Reliability

9. The need for HPC on the electrical grid • Parallel Computing is essential • Only explicitly parallelized algorithms can take advantage of multi-core parallel computers • Parallel Computing is “an art” • Parallelization approaches are problem-dependent • Computing implementation needs to consider “good match” of computing architecture and the problems

10. Normal operations • Asset underutilization • Limited market opportunities • Lead to emergency operations • Emergency operations • Blackouts and cascading failures slow static Operator SCADA ~ seconds Violations State Estimation Contingency Analysis ~ minutes ~ minutes Off-line Transient/Voltage Stability Analysis Constrained solutions Market Operation Ratings & Limits ~ hours seasonal Today’s Electrical Grid Operations Paradigm

11. Trends Impacting Control System Security • Open Protocols • Open industry standard protocols are replacing vendor-specific proprietary communication protocols • General Purpose Computing Equipment and Software • Standardized computational platforms increasingly used to support control system applications • Interconnected to Other Systems • Connections with enterprise networksto obtain productivity improvementsand information sharing • Reliance on External Communications • Increasing use of public telecommunicationsystems, the Internet, and wireless for controlsystem communications • Increased Capability of Field Equipment • “Smart” sensors and controls with enhancedcapability and functionality

12. The Emerging Cyber Threat • Industry has long history of planning for and coping with natural disasters and other reliability events • Through industry standard operating procedures, there is much effort expended to reduce likelihood of cascading outages leading to widespread blackouts • Historically, cyber security focused on countering unstructured adversaries • e.g., individuals, untargeted malicious software, human error • Very little protection against structured adversaries intent on exploiting vulnerabilities to maximize consequences • e.g., terrorist groups, organized crime, nation states • Insider threat remains very challenging, can be used as part of structured threat vector • New possibilities for widespread sustained outages resulting from cyber attack are now being contemplated • But industry still not ready to cope with this threat

13. The need for HPC: State Estimation • Power system State Estimation (PSE) • Given: power grid topological information, telemetry on line flows, bus injections or bus voltages • Compute: a reliable estimate of the system state (bus voltages), validate model structure and parameter values • Calculated using Weighted Least-Squares (WLS) method • WLS: minimize • Where r = z - h(x), and r is the residual vector, x is the system state, z is a vector of measured quantities, h is a vector function, wi is the weight for residual ri and W is as diagonal matrix. • This is a non-linear problem, which is solved using the Newton-Raphson iterative procedure

14. The need for HPC: State Estimation • PSE • Every iteration of the method requires solving a large set of sparse linear equations • Sparse matrices are derived from the topology of the power grid being analyzed • The number of non-zeros per row varies greatly and the matrix is badly conditioned • The set of linear equations can be solved using direct solvers such as sparse LU factorization or iterative solvers such as sparse Conjugate Gradient (CG) • PSE is a critical element of the software used by power grid control centers • Under real-time constraints (< 10 seconds) • Commercial PSE solvers are not commonly parallel

15. Power System State Estimation Situational Awareness? August 14, 2003 Blackout August 13, 2003 Normal Do we know what really happened? Could it be prevented? Source: NOAA/DMSP

16. Model Validation Need and Challenges Recorded system dynamics vs. simulation results: California and Oregon Intertie (COI) real power flow during the August 10, 1996 event

17. The need for HPC: Multithreaded Platforms for Contingency Analysis • Growing class of scientific applications is becoming memory-bound • Many scientific applications exhibit irregular memory access patterns • CPU and memory technology trends indicate that the situation will not improve anytime soon • Multithreaded architectures offer an appealing alternative for irregular applications • Processors tolerate memory access latencies by switching execution context between multiple hardware threads • Examples of such architectures are the Cray MTA-2 and XMT systems and the Sun Niagara • Latency tolerance mechanisms should improve the performance of irregular, data-intensive applications with abundant fine-grained parallelism

18. Role of Contingency Analysis • From “N-1” to “N-x” • To improve situational awareness • From Balancing Authorities to a Wide Area • Example: 35 BAs in west • Further require “N-x” CA • To better understand cascading failures • N-x Contingency Analysis • Result in a large number of cases. “N-5”  1020 cases for the west =~ 1020 seconds + lots of data • Needs: better contingency selection and post-processing

19. Looking forward: PNNL’s vision of advanced contingency analysis Industry Need: Multi-failure (N-x) massive contingency analysis Issue: Massive number of cases Massive amount of data Need: Smart case selection Operator-oriented data processing Our Solution: Contingency Analysis Data 10Z cases 10Y cases 10X cases Prediction Major Tasks: past now future Trending Time Graph Theory Contingency Index Sorting Visual Analytics & Graph Trending Expected Outcomes: Interactive Graphing • An advanced tool for contingency analysis to support the operation of the nation’s critical infrastructures, e.g. power grids • Analysis capabilities through generic implementation of algorithms on Cray XMT ThreadStorm processors, applicable to other problems

20. Tool Suites for Advanced Power System Simulation SCADA Measurements Phasor Measurements Parallel state estimation Dynamic state estimation Parallel contingency analysis On-line voltage stability Look-ahead dynamic simulation Market Monitoring Dyn. contingency analysis Visualization  National Level:monitoring, planning, design  Utility Level Applicable to current grid operations  Next-generation grid operation tools

21. Normal operations • Optimized asset utilization • Enabled regular & ancillary markets • Predict emergencies • Emergency operations • Prevent/mitigate failures fast dynamic SCADA & Phasor Operator Dynamic State Estimation Dynamic CA, On-line TSA/VSA ~ (sub)seconds Violations ~ (sub)seconds ~ (sub)seconds Dynamic Ratings & Limits Real-Time Optimized Market Operation Constrained solutions < 1 hour Looking forward: PNNL’s long-term vision for electrical grid operations

22. Electricity Infrastructure Operational Center at PNNL • Use HPC to analyze sensor data for real-time grid monitoring, prediction, and operation: data-driven models, stochastic simulations, visual analytic tools, secure sensor network infrastructure, data provenance

23. QUESTIONS?