Reliability Coordination Task Force Report to Finance and Audit Committee December 5, 2012

1. Presented by Members:Tom Botello - Southern California EdisonTerry Baker - Platte River Power Authority Reliability Coordination Task Force Report to Finance and Audit Committee December 5, 2012

2. Report Agenda • FAC Assignment • Summary of the Issue/Finding/Conclusions • Real-time Operations • Operations Support • Western Interconnection Synchrophasor Program (WISP) • Compliance • Training • Recommendations

3. Reliability Coordination Task Force (RCTF) Members • Terry Baker • Tom Botello • John Cupparo • Graham Edwards • John Meyer

4. RCTF 2012 Working Sessions • July 18 Kick-off meeting in Portland, Oregon • August 15 In-person meeting in Denver • August 24 Meeting by Webinar • October 22 In-person meeting at the Vancouver RC • November 20 Meeting by Webinar • November 23 Meeting by Webinar • November 27 Meeting by Webinar • December 4 In-person meeting in Salt Lake City

5. FAC Assignment • RCTF to examine the following: • Original scope of RC duties and expenses • Current scope of duties including expanded duties since 2009 • Future scope of duties and expenses • Analyze key drivers for operational and budgetary change, WECC’s response to those drivers, and provide recommendations

6. Existing WECC RC Functions and Current Staffing Real-time Operations Operations Support Compliance WISP Training Other Operations Planning • Current -day/Real-time Support • Next-day Study Data Validation and Mapping • Seasonal SOL Coordination and Validation • Emergency Plan Coordination • Monitor and analyze real-time system conditions • Coordinate system restoration activities • Proactively respond to and mitigate potential adverse impacts to the BES • Communicate with and direct corrective actions of registered entities during actual SOL exceedances • Identify and mitigate potential/actual IROLs in real-time operations horizon • Application Architects • Application Support • DTS Support • EMS Modeling • RAS Engineer • Architecture Support • Communications • IT Infrastructure • Systems Administrator • Document and Evidence Management • Audit Preparation • Event Analysis • Standards Development • Compliance Reporting • Disturbance Reporting • Broaden deployment of phasor measurement units throughout the Western Interconnection • Implement telecommunication network necessary to support a large-scale production synchrophasor system • Install synchrophasor applications for Western Interconnection • Increase ability to integrate variable generation   • Enhance power system analysis capability/model validation • Develop/Deliver in-house training to new RC trainees and current RC staff to ensure compliance with mandatory standards and retention of NERC RC System Operator certification • Conduct just-in-time training for new tools, applications, procedures, and processes • Update job task analysis • Develop/deliver NERC-certified restoration training to BAs, TOPs, and GOs • Direction and Oversight • Administrative Support • WECC and NERC Committee Work • WECCRC.org Development • Membership Request Management Totals Current FTE – Filled Current FTE Approved – Unfilled 82 16

7. Summary of the Issue • FERC/NERC Requirements • Highest level of authority responsible for the reliable operation of the BES • Authority to prevent or mitigate emergencies in next-day analysis/real-time • Wide Area view of BES (situational awareness) • Tools, processes, and procedures • Broad purview to enable IROL calculation(s) that may be based on parameters of transmission systems beyond any TOPs vision

8. Summary of the Issue • Reliability Coordination in the Western Interconnection • Past – Reliability through collaboration • Hosted by utilities – CAISO, BPA, WAPA • Use of facilities, tools, and personnel • Present – WECC’s Responsibility • Independent facilities, tools, and responsibilities • Compliance obligations questioned • Reliability vulnerabilities exist • Future – Implement RCTF Recommendations • Increase staffing in critical areas • Meet regulatory compliance requirements • Reduce reliability vulnerabilities

9. Summary of RCTF Findings • Additional personnel are recommended in the following areas: • Real-time operations • Broad purview to enable IROL calculations (granularity, specific focus) • Additional real-time tools, processes, procedures • Operations support • Creation of operating plans, procedures (seasonal and next-day studies) • System Modeling and model validation • IT application support requirements • Compliance support • Training

10. Summary Conclusions • WECC has a high level of responsibility and is at risk for reliability and compliance issues • Staffing is a major factor • Tools are a major factor • Emergent tools and Standards will add to these risks • WISP-related displays, data, training • PER-005 and EOP-005

11. Real-Time Operations • RC Contingency Analysis • Four RC System Operators on shift on September 8, 2011 • Prior to loss of North Gila-Hassayampa 500-kV line, there were 287 harmful contingencies identified by RTCA • RTCA run following line loss identified 632 harmful contingencies • Between start of event and SONGS units tripping (less than 11 minutes) 1,185 alarms were generated

12. Real-Time Operations • Similar events to SW Outage reveal RC real- time staffing concerns • Ability to monitor current system specifics questioned by regulators • Increase in required monitoring of RAS • Emergent tools such as WISP • Sub-100-kV facilities • Specific interconnection area knowledge by RC System Operators needed to meet requirements • RC phone calls per quarter have increased to over 37,000

13. Real-Time Operations

15. Vancouver RC Total Load: 104506 MW Total BAs: 21 Total TOPs: 29 Loveland RC Total Load: 48888 MW Total BAs: 16 Total TOPs: 25

16. Vancouver RC Total Load: 49666 MW Total BAs: 16 Total TOPs: 17 CA/MX RC Total Load: 54840 MW Total BAs: 6 Total TOPs: 13 Loveland RC Total Load: 47888 MW Total BAs: 15 Total TOPs: 24

17. Real-Time Operations - Documentation • The RC must create and maintain operating plans, procedures, and processes • This documentation must: • Identify RC System Operator actions taken to address specific system conditions and events • Reflect compliance to mandatory NERC, FERC, and Regional requirements, policies, and procedures • Currently no specific position to perform this function

18. Real-Time Operations − Findings • Additional personnel are recommended in order to: • Monitor the Western Interconnection to the level of granularity expected by FERC, NERC, and the NPCC • Respond effectively to the volume of alarms and RTCA results during an event to determine appropriate corrective action • Create and maintain Operating Plans, Procedures, and Processes as required by NERC Standards

19. WECC RC Function and Recommended FTE Increase 33 0 14 Real-Time Operations • Monitor and analyze real-time system conditions • Coordinate system restoration activities • Proactively respond to and mitigate potential adverse impacts to the BES • Communicate with and direct corrective actions of registered entities during actual SOL exceedances • Identify and mitigate potential/actual IROLs in real-time operations horizon Current FTE – Filled Current FTE Approved – Unfilled Future FTE Projection FTE Totals

20. Operational Planning (Seasonal) • SW Outage event and regulatory findings revealed seasonal planning gaps • Not all WECC paths studied seasonally • TOPs coordination weak • Incomplete WECC contingency analysis • Mitigating reliability vulnerabilities • With an enhanced seasonal process and more thorough assessments, the event may have been predicted

21. Operational Planning (Next-Day) • Why did the RC next-day study not see the problem that caused SW Outage event? • Only study next-day peak • Study automation incomplete • Need to assess sub-100-kV facilities • Forecast data quality inputs need improvement and are too numerous • Number of scheduled outages continues to increase • A million rows of unit commitment data daily

22. Operations Planning - RCTF Findings • Additional personnel are recommended to perform the following activities: • Seasonal Planning: • Take lead in implementation of an enhanced process with more thorough assessments • Participate in the resolution of identified gaps in the WECC Regional process • Next-Day Study: • Complete study automation • Assess sub-100-kV facilities • Ensure quality of data inputs

23. WECC RC Function and Recommended FTE Increase 5 9 11 Operations Planning • Current -day/Real-time Support • Next-day Study Data Validation and Mapping • Seasonal SOL Coordination and Validation • Emergency Plan Coordination Current FTE – Filled Current FTE Approved – Unfilled Future FTE Projection FTE Totals

24. Operations Support – System Modeling • Model Impacts to Situational Awareness • Three ways to improve accuracy of RTCA: • Increase accuracy of facility ratings and SOLs • Increase number of RAS modeled • Expand modeling of lower-voltage networks that impact the BES • Sub-100-kV systems that parallel higher voltage networks • Sub-100-kV systems with both automatic and manually-switched reactive devices

25. Operations Support – System Modeling • RAS Engineer and EMS Modeling – • WECC RC System Operators are required to improve awareness of the impacts of RAS • Continuing increase in the number of WECC RAS • A key component of improved awareness – • RAS modeling in the EMS and real-time contingency analysis (RTCA) Only 45 of 206 RAS modeled in RTCA

26. Operations Support – System Modeling

27. Operations Support – System Modeling

28. Operations Support – System Modeling • Addressing Model Deficiencies • WECC RC needs to be more proactive in the modeling process by increasing the frequency of model reviews with TOPs • West-wide System Model and Base Case Reconciliation Task Force (WBRTF) • Develop process for periodic review and sign off of model accuracy • Add more power system measurements • Share data and application results

29. Operations Support - Applications and IT • Applications and IT areas of responsibility include, but are not limited to: • coordinated outage scheduling • EMS system support • load and interchange forecasting • operating system maintenance • change management • cyber security administration • networking

30. Operations Support - Applications and IT • Application Architects and Support areas of responsibility include, but are not limited to: • support planning of IT initiatives to evaluate and determine new tools, technology, and integration with external drivers such as SAFNR or data requests • attend working groups • review new software releases • set system direction • upgrade paths • ensure a timely response for application support • ensure continued compliance with new CIP Standards

31. Operations Support - Applications and IT • IT and Applications increases since 2009: • Number of applications increased from seven to 16 • Number of users increased from 15 to 34 • As a cost savings measure, RC tool applications have been developed in-house • Creating a burden: volume of custom applications requiring on-going maintenance have increased • Number of developed applications increased from two to 53 • Support staff increased from five to six

32. Operations Support - IT Infrastructure

33. Operations Support - IT Infrastructure • In 2009 average was 40 servers/IT employee • Today, it is over 112 servers/IT employee

34. WISP − IT Infrastructure • WISP is cutting edge technology • WECC staff to develop and maintain custom software for the WISP program • There will be a 40 percent increase in the amount of data storage in SQL databases with the synchrophasor systems • Voltage Stability Analysis will be integrated with 1500 points of synchrophasor data that will be maintained and tuned

35. Operations Support − RCTF Findings • Additional personnel needed in the following in order to: • Identify and address model deficiencies • Make necessary improvements to the model • Ensure proper model validation with TOPs • Model all WECC Region RAS and real-time RAS data points in the WECC RC EMS • Provide maintenance and support for an increasing number of users, applications, and servers • Ensure compliance

36. WECC RC Function and Recommended FTE Increase 3 20 12 • Operations Support • Application Architects • Application Support • DTS Support • EMS Modeling • RAS Engineer • Architecture Support • Communications • IT Infrastructure • Systems Administrator Current FTE – Filled Current FTE Approved – Unfilled Future FTE Projection FTE Totals

37. Compliance • WECC RC Compliance staffing has experienced high turnover in the last three years • As of 11/26/12 only one staff position is dedicated to Compliance • Compliance monitoring and documentation undertaken by upper-level RC management resulting in additional work often outside normal duty hours

38. Compliance - RCTF Findings • The WECC RC increased staffing will improve the ability to: • Meet the March 2012 WECC Board Resolution that established clear compliance expectations for WECC • Meet increased compliance and reliability responsibilities • Meet increased regulator expectations • Mitigate increasing regulatory and financial risks

39. WECC RC Function and Recommended FTE Increase 1 1 1 Compliance • Document and Evidence Management • Audit Preparation • Event Analysis • Standards Development • Compliance Reporting • Disturbance Reporting Current FTE – Filled Current FTE Approved – Unfilled Future FTE Projection FTE Totals

40. Training • Reliability Coordination Trainers • Training responsibilities are identified in NERC Reliability Standards • DTS Engineers • Support increase in system restoration drills, exercises, and simulations • Assist in developing implementation of new TOP and region-specific restoration training scenarios

41. Internal -Training Hours NOTE: Includes projected hours for remainder of 2012

42. Training - RCTF Findings • Additional personnel recommended to: • Provide required training to RC staff • Provide required training to regional reliability entities • Ensure training documentation is maintained as a provider • Ensure compliance with NERC Reliability Standards

43. WECC RC Function and Recommended FTE Increase 6 4 5 Training • Develop/Deliver in-house training to new RC trainees and current RC staff to ensure compliance with mandatory standards and retention of NERC RC System Operator certification • Conduct just-in-time training for new tools, applications, procedures, and processes • Update job task analysis • Develop/deliver NERC certified restoration training to BAs, TOPs, and GOs Current FTE – Filled Current FTE Approved – Unfilled Future FTE Projection FTE Totals

44. Conclusions • WECC has a high level of responsibility and is at risk for reliability and compliance issues • Staffing is a major factor • Tools are a major factor • Emergent tools and Standards will add to these risks • WISP related displays, data, training • PER-005 and EOP-005

45. RCTF Recommendations • Increase monitoring areas from two to three • Add 47 WECC RC FTEs to enhance and improve the capability of the RC as follows: • 2013 - 17 FTEs • 2014 - 22 FTEs • 2015 - 8 FTEs • Acquire additional tools as identified in the projected budget

46. WECC RC Function and Recommended FTE Increase 9 2 0 33 0 14 20 3 12 1 1 1 6 5 4 9 5 11 Real-time Operations IT Applications, Engineering Support, & IT Support Compliance WISP Training Other Operations Planning • Current-day/ real-time Support • Next-day Study Data Validation and Mapping • Seasonal SOL Coordination and Validation • Emergency Plan Coordination • Monitor and analyze real-time system conditions • Coordinate system restoration activities • Proactively respond to and mitigate potential adverse impacts to the BES • Communicate with and direct corrective actions of registered entities during actual SOL exceedances • Identify and mitigate potential/actual IROLs in real-time operations horizon • Application Architects • Application Support • DTS Support • EMS Modeling • RAS Engineer • Architecture Support • Communications • IT Infrastructure • Systems Administrator • Document and Evidence Management • Audit Preparation • Event Analysis • Standards Development • Compliance Reporting • Disturbance Reporting • Broaden deployment of phasor measurement units throughout the Western Interconnection • Implement telecommunication network necessary to support a large-scale production synchrophasor system • Install synchrophasor applications for Western Interconnection • Increase ability to integrate variable generation   • Enhance power system analysis capability/model validation • Develop/Deliver in-house training to new RC trainees and current RC staff to ensure compliance with mandatory standards and retention of NERC RC System Operator certification • Conduct just-in-time training for new tools, applications, procedures, and processes • Update job task analysis • Develop/deliver NERC certified restoration training to BAs, TOPs, and GOs • Direction and Oversight • Administrative Support • WECC and NERC Committee Work • WECCRC.org Development • Membership Request Management FTE Totals FTE Totals FTE Totals FTE Totals FTE Totals FTE Totals FTE Totals 4 0 5 Totals Current FTE – Filled Current FTE Approved – Unfilled Future FTE Projection 82 16 47

47. Budget Projections

48. Questions?

49. Back Up Slides

50. Control Room Facilities Comparison MISO WECC RC Vancouver