Data Extracts & ReportingRecent Issues ERCOT Information Technology Data Extracts Working Group 11/27/07
Agenda • TML API Outage 9/29/07-10/1/07 • Market Notice Review • M-A103107-06 • R-A110907-01 • M-A111207-02 • M-C111407-02 • M-A112107-03 • Update regarding the search for a new EIS manager
TML API Issue • Issue: Market connectivity issues to TML via the API • Issue Dates: 9/29/07 – 10/1/07 • Market Notice: Issue was not experienced market-wide. • Root Cause: A greater than allowed number of connections to the API caused the Enterprise Application Integration (EAI) layer to refuse some connection requests. • Resolution: Increasing the number of connection threads in the EAI layer allowed all requests to be handled appropriately. Monitoring: ERCOT has added additional monitoring to capture and correct this prior to becoming a market impacting issue.
10/29/07 - 11/26/07 Extract Issues • Issue: ERCOT experienced delays with posting several data extracts • Notice Dates: 10/31/07-11/9/07 • Market Notice: M-A103107-06 • Root Cause: ERCOT batch process delays due to Oracle bug when running database statistics. • Resolution: Utilize Oracle recommended workaround for running database statistics. • Further information and timeline on next slide
10/29/07 - 11/26/07 Extract Issues Market Notice: M-A103107-06 (cont’d) Incident Timeline • 10/21 – Server consolidation performed during maintenance outage • 10/27 – Database statistics fail – first time running statistics on new platform, causes batch process to run slowly • 10/28 – Daylight Savings Time adjustment issue with batch scheduling application • 10/29 – Partial rollback of database statistics • Severity 1 Technical Assistance Request opened with Oracle • Batch process continuing to run slowly • 10/30 – Events • 0615 – decision made to rerun database statistics, batch held from 0745 until 1030 when statistics estimated to have completed • 1030 – reinstated batch while statistics were still refreshing • 1915 – terminated statistics refresh, attempted statistics with smaller database sample • 1945 – full rollback to old statistics – batch performs at expected levels • Also began making adjustments to batch schedule in attempt to make up time which caused issue described in Market Notice M-C111407-01 • 11/5 – Estimated catch up date • 11/9 – Actual catch up date • 11/16 – Oracle confirms issue that caused statistics refresh to run for so long was due to a software bug
10/29/07 - 11/26/07 Extract Issues • Issue: Additional ESI ID Extract to post from some Market Participants • Notice Date: 11/9/07 • Market Notice: R-A110907-01 Extracts - Retail • Root Cause: • The 727 delete objects (esiidservicehist_delete, lschannelcutheader_delete, esiidusage_delete) only pull rows that have been actually deleted. The extract views explicitly only pull rows that have an ov_row_status of 'ID' or 'UD‘. • Resolution: • Removal of the " and ov_row_status in ('ID' or 'UD') " criterion from the delete extract views. To correct the missed rows, a supplemental extract is created.
10/29/07 - 11/26/07 Extract Issues • Issue: ERCOT is experiencing delays with posting several extracts • Notice Date: 11/12/07-11/13/07 • Market Notice: M-A111207-02 • Root Cause: • Complications with the batch scheduling software encountered while completing an ERCOT requested rerun of the 11/8 batch (10/31 trade date) resulted in the 11/8 batch running on Sunday 11/10 and completing on 11/11. We moved forward with the 11/9 batch and it completed at approximately 5:00 p.m. on Saturday. When Lodestar batch ran out of order, the controls table was not accurately populated. • Resolution: • The control table rows were manually entered. A SIR has been requested to modify the control record creation process.
10/29/07 - 11/26/07 Extract Issues • Issue: Data for operating day 10/31 was not included in the Load Generation, Settlements and Billing extracts posted on 11/12 • Notice Date: 11/14/07 – 11/15/07 • Market Notice: M-C111407-02 • Root Cause: • Complications with the batch scheduling software encountered while completing an ERCOT requested rerun of the 11/8 batch (10/31 trade date) resulted in the 11/8 batch running on Sunday 11/10 and completing on 11/11. We moved forward with the 11/9 batch and it completed at approximately 5:00 p.m. on Saturday. When Lodestar batch ran out of order, the controls table was not accurately populated. • Resolution: • Control table rows previously manually entered did not take into account that the 11/8 batch day data was not complete until 11/11. Related to market notice M-A111207-02 . A SIR has been requested to modify the control record creation process.
10/29/07 - 11/26/07 Extract Issues • Issue: Delays with posting several data extracts/reports • Notice Date: 11/21/07 - present • Market Notice: M-A112107-03 • Root Cause: • Extracts are posting one day behind due to batch run delays. • Resolution: • Currently analyzing several key areas including correcting an issue with backups and tuning database performance.