Maintenance Strategies with Limited Resources

1. Maintenance Strategies with Limited Resources

2. Goal • We will learn if you have reliable electrical power systems • Why that is important • What to do if you don’t • Learn a financial method to rank alternative methods of improving reliability

3. Agenda • Where We Are Today • What Can Be Done? • IEEE 493 Process (Identify Candidates) • Predicting Failures • Economic Analysis / Prioritization • Solutions Available

4. Where We Are Today

5. Where We Are Today • Fact # 1: The aggregate economic loss of electrical power disruptions has climbed to more than $100 billion per year or more than 1% of U.S. Gross Domestic Product! • Recent events have demonstrated the fragility of our aging power grid. With transmission networks operating close to their stability limits, minor faults can cause cascading outages. Capacity limitations in several regions can lead to economic losses that cascade through the economy, causing loses for not only residential, but also commercial and industrial institutions.

6. Where We Are Today • Fact # 2: The recent power outages have been in the works for the last several years! • U.S utilities have always taken pride in their uptime and system performance. Over the past several years, due to industry-wide deregulation, market pressures for rate reductions, business restructuring and downsizing, overall investment in infrastructure has not been at traditional levels. The decoupling of transmission, distribution and generation has caused disruption in traditional business models and industry workings (i.e. the vertically integrated utility no longer exists in deregulated markets)

7. Where We Are Today • Fact # 3: De-regulation has contributed to loss of stability. • When the Federal Energy Regulatory Commission dictated that the electrical transmission network was to be opened to the free market, it allowed anyone to transmit power over any transmission line. This allowed generators outside a customer’s service area to bid on a distant customer’s power requirements and be guaranteed access to that customer over the transmission system. As a result, the owners of the transmission lines lost some of their ability to maintain stability since these lines now carry power generated outside their control. As demand has continued to increase at an average 2% per year and because few will accept new transmission lines in their backyard, grid stability continues to degrade over time.

8. Where We Are Today • Fact # 4: The 2000 dotcom implosion and the resultant relaxation of electrical demand has temporarily relaxed the demands placed on electrical transmission system, but the problem remains. • The recent retraction of the economy has reduced electrical demand temporarily. However little or no new transmission was constructed during that economic downturn. Electrical demand is again returning to historic levels. We can only expect the problem to return and even become worse as the economy expands to more robust levels.

9. Conclusion • This problem will worsen before it improves • Universities must take action themselves

10. Agenda • Where We Are Today • What Can Be Done? • IEEE 493 Process (Identify Candidates) • Predicting Failures • Economic Analysis / Prioritization • Solutions Available

11. What Can Be Done? A. Protect Yourself From External Problems • Install Local Backup Power • Natural Gas, LPG, Diesel from 1 kW to over 10 MW • Install Voltage Correction • SAG Correction Equipment • UPS • Capacitors • Engineering Review • Audits • Site Supervision • Equipment Commissioning • Turnkey Installation

12. What Can Be Done? • Many university outages are not caused by loss of utility power, but rather by internal problems • Protect yourself from internal problems • Equipment failure, accelerated by: • Dust, dirt, moisture, rodents, etc. • Thermal cycling, vibration induced loosening, etc. • Obstruction of ventilation, etc. • Operator error • Reduction in funding for preventative maintenance • But when resources are tight, where should they be spent to give maximum uptime?

13. What Can Be Done? B. Protect Yourself From Internal Problems • Bring in an expert • Reliability Study • Site Survey (e.g. evaluation of on-site generation, UPS, calculation of reliability of existing system, etc.) • Thermography • Coordination Study • Harmonics Study • Install Predictive Diagnostic equipment • Early warning of pending failure in MV Equipment

14. Agenda • Where We Are Today • What Can Be Done? • IEEE 493 Process (Identify Candidates) • Predicting Failures • Economic Analysis / Prioritization • Solutions Available

15. IEEE 493 Process 1. Establish Current Condition of Facility 2. Determine Likelihood of Serious Problem Based on this Condition 3. Sort to Find Equipment Most at Risk to Cause Problems 4. Identify the Predictive Techniques that Gives Early Warning of Problems at that Equipment

16. What Is “Current Condition?” • The ‘Quiet Crisis’Term created by Paul Hubbel, Deputy Director, Facilities and Services, Marine Corps. Government Executive Magazine, Sept 2002.When he was asked “why isn’t preventative maintenance adhered to more closely in government facilities?”“We call it the ‘quiet crisis’ because a lot of maintenance problems take time to occur and are not noticed until damage occurs”.

17. Current Condition? • This leader says many follow the “fix when broken” but not before mentality? • What happens if thepower goes out at yourfacility for an extendedperiod of time?For example, we would expect thatfacilities (such as universities & hospitals) won’t have power outage problems!

18. 06-AUG-03 Apower failure forced New York University Medical Center to shut its emergency room and turn away visitors and some patients yesterday, as the hospital staff struggled with limited use of the air-conditioning, computers and other equipment. Chris Olert, a spokesman for Consolidated Edison, said, "We know that it was a combined failure of some of our equipment and some of the hospital's equipment, but we don't know exactly what triggered it.“ He said a cable feeding electricity to the medical center was damaged and had to be bypassed. When the power failed, the hospital's backup generators automatically turned on, but they could not carry the entire load, so hospital officials shut down some functions to preserve electricity for the most crucial ones. "No critical services were affected," said Lynn Odell, a hospital spokeswoman. But hospital employees interviewed outside the building and family members of some patients said things were seriously disrupted for a time. One worker told of a darkened pharmacy with dormant computers, where pharmacists using flashlights filled out paperwork by hand and responded to orders by telephone rather than computer. Another spoke of a stiflingly hot surgery department where some medicines spoiled in a nonworking refrigerator. (New York Times 8-6-2003)

19. Fairview Southdale Hospital, MN 21-SEP-02 …in the case of Fairview Southdale Hospital's power failure, which lasted an hour, a power line wound up carrying more than its normal load for four days, which melted a fuse. That set off a chain of events that wound up overloading the hospital's emergency generators, causing them to fail, too. Mark Enger, the hospital's president, said the power failure could have been life-threatening. The power dropped but didn't go out entirely. It dropped enough, however, to trigger the hospital's three emergency generators. But because of the way the system is wired, the generators' cooling fans failed to work, the generators overheated and the hospital lost all power at about 11 a.m. Enger said there were eight surgeries going when the power failed. Four of the surgeons were able to finish their operations, while the other four finished quickly and re-scheduled the surgeries for the next day. Partial power was restored by noon, and full power was restored the next day, Enger said. Xcel is adamant that its maintenance practices are not posing widespread service problems. But one NSP worker said the hospital's power failure is but a symptom of the electrical grid's ill health. "Their work force is stretched so thin, they're putting people in jeopardy," he said.. (St. Paul MN, Pioneer Press 8-6-2003)

20. Rhode Island Hospital On April 16, 2002, Rhode Island Hospital and Women & Infants Hospital lost power for an afternoon, leaving various parts of the 33-building campus dark for varying amounts of time. At Women & Infants, an emergency generator immediately turned on, emergency lights came on, and no essential services were disrupted, according to a Providence Journal article. However, at Rhode Island Hospital, the backup electric system did not work, prompting Providence Mayor Vincent A. Cianci Jr., to tell the Providence Journal, “A hospital of this magnitude and this size should not have these problems.” Surprisingly, power outages do happen with alarming frequency to big hospitals. In fact, they’ve happened at Rhode Island Hospital campus before. In September 1999, a blackout plunged the entire campus into darkness. The backup systems failed once again and this time a patient died after his respirator failed. Then, in January 2000, another power failure forced the hospital to rely on backup generators for nearly two hours and shut down nonessential equipment and lights. A faulty ceramic insulator at a substation on the hospital campus caused the failure. Then a damaged coil prevented some of the backup power from flowing back into one of the hospital buildings. (EC&M Magazine 8-1-2002)

21. Current Condition? • Why would there be such problems in critical tested systems? • Budget Cuts / Management Redirection of Maintenance Funds? • This results in “Crisis Mode Operation” or “Fix What’s Broken and Skip the Rest” mentality • But if you operate this way, how do you guess what will break next and where money should be targeted? • Is there an analytical way of targeting scarce resources?

22. IEEE 493 Step 1 1. Establish Current Condition of Facility 2. Determine Likelihood of Serious Problem Based on this Condition 3. Sort to Find Equipment Most at Risk to Cause Problems 4. Identify the Predictive Techniques that Gives Early Warning of Problems at that Equipment

23. IEEE 493 Step 1 • What is the likelihood of a loss of MV power at a particular point in the university? • How many sources of supply • Are there one or more single-points-of-failure? • What is the likelihood of those points failing? • Use algebra to combine probabilities

24. IEEE 493 Step • For example, if power flows from utility like this: Utility Switch Breaker Load

25. IEEE 493 Step • For example, if power flows from utility like this: Utility Switch Breaker Load 99.9% 99.99% 99.99%

26. IEEE 493 Step • For example, if power flows from utility as below: Utility Switch Breaker Load 99.9% (8.7 hr/yr) x 99.99% (0.87 hr/yr) x 99.99% (0.87 hr/yr) = 99.88% (10.5 hr/yr) • Overall reliability is poorer than any component reliability

27. Finding “Downtime Per Year” • Downtime / Year • Made up of two components • How often failures occur • Measured as Mean-Time-Between-Failures • MTBF • e.g. 80000 hours • How long a failure knocks you out • Measured as Mean-Time-To-Repair • MTTR • e.g. 8 hours

28. Finding “Downtime Per Year” • Multiply together to determine downtime-per-year • How often : e.g.. 1 failure every 80000 hours • How long : e.g.. 8 hours each failure 8 8 hrs length of outages (MTTR) hours 0 1 2 3 4 5 6 7 8 9 9 9 80008 hrs frequency of outage (MTBF) years 80000 hrs

29. Finding “Downtime Per Year” • Multiply together to determine downtime-per-year • How often : e.g.. 1 failure every 80000 hours • How long : e.g.. 8 hours each failure 8 8 hrs length of outages (MTTR) hours How thick is this lower line, if we take the same probability and spread it over 80000 hours? 0 1 2 3 4 5 6 7 8 9 frequency of outage (MTBF) years 80000 hrs

30. Finding “Downtime Per Year” • Multiply together to determine downtime-per-year • How often : e.g.. 1 failure every 80000 hours • How long : e.g.. 8 hours each failure

31. IEEE 493 Step 2 1. Establish Current Condition of Facility 2. Determine Likelihood of Serious Problem Based on this Condition 3. Sort to Find Equipment Most at Risk to Cause Problems 4. Identify the Predictive Techniques that Gives Early Warning of Problems at that Equipment

32. IEEE 493 • Likelihood of failure of electrical equipment • Low • Consequences of failure of electrical equipment • High

33. Substation Transformer

34. Substation Transformer

35. Consequences of Failure • After certain failures (such as that transformer failure), power cannot be restored for many hours • Because of the large amount of energy within electrical equipment, that equipment can fail explosively • This may cause mechanical damage which limits ability to quickly repair and restore equipment to service

36. Equipment Destruction

37. Forensic Analysis of Equipment Destruction

38. Equipment Destruction

39. Worker Injury

40. Equipment Destruction

41. Equipment Destruction

42. Equipment Destruction

43. Consequences • This equipment isn’t going to be repaired in a few hours, or even a few days.

44. IEEE 493 • Likelihood (Failures / yr) • Low • Consequences (Hrs / failure) • High

45. IEEE 493-1997 (Gold Book) Analysis IEEE Std 493-1997, Table 7-1 Low likelihood High consequences Note: could be higher, but protective relays are usually installed in draw-out cases, so the delay in repairing is due to: 1) diagnosing the problem 2) locating a spare 3) installing a spare 4) reprogramming setpoints to match previous unit 5) restarting system

46. IEEE 493-1997 (Gold Book) Analysis IEEE Std 493-1997, Table 7-1 Failures/yr * Hours/Failure = Hours/Yr

47. IEEE 493-1997 (Gold Book) Analysis IEEE Std 493-1997, Table 7-1 Failures/yr * Hours/Failure = Hours/Yr * when no on-site spare is available ** below ground *** 3 connected to 3 breakers

48. IEEE 493 Step 1 52 52 52 52 52 • Example: • Likelihood of failure at an ICU = • f(SW1) x f(CBL1) x f(TX1) x f(CBL2) + f(BKR1) x f(RLY1) x f(BUS1) x f(BKR5) x f(RLY5) x f(CBL6) • f(…) means probability of failure of that component SW1 CBL1 TX1 CBL2 BKR1 RLY1 51 BUS1 BKR2 BKR3 BKR4 BKR5 51 51 51 51 RLY2 RLY3 RLY4 RLY5 CBL3 CBL4 CBL5 CBL6 ICU

49. ICU Failure Scenario SW1 CBL1 52 52 TX1 CBL2 BKR1 RLY1 51 BUS1 BKR5 51 RLY5 CBL6 f(Utility) + f(SW1) + f(CBL1)+ f(TX1)+ f(CBL2) + f(BKR1)+ f(RLY1)+ f(BUS1) + f(BKR5) + f(RLY5) + f(CBL6) • f(Utility) = 8.76 hrs/yr (99.9% ) • f(SW1) = .022 hrs/yr (99.9997% • f(CBL1300ft) = 300/1000 * 0.1624 = 0.049 hrs/yr (99.9994%) • f(TX1)= 1.026 hrs/yr (99.988%) • f(CBL2100ft) = 100/1000 * 0.1624 = 0.0162 hrs/yr (99.9998%) • f(BKR1), (BKR5) = .2992 hrs/yr (99.9966%) • f(BUS1) = .2733 hrs/yr (99.9969%) • f(RLY1), (RLY5) = .001 hrs/yr (99.999989%) • f(CBL6300ft) = 300/1000 * 0.1624 = 0.049 hrs/yr (99.9994%) • Total = 99.9%Utility x 99.9997%SW1 x 99.9994%CBL1 x 99.988%TX1 x99.9998%CBL2 x 99.9966%BKR1 x 99.999989%RLY1 x 99.9969%BUS1 x99.9966%BKR5 x 99.999989%RLY5 x 99.9994%CBL6= 99.88% (10.8 hrs/yr)

50. ICU Failure Scenario • Obviously, this is completely unacceptable reliability, but our example isn’t exactly accurate • An ICU will always be connected to a critical backup bus fed from a generator • It may or may not have redundant transformers • How do we recalculate with a generator and transfer switch in the system?