erhtjhtyhy

1. erhtjhtyhy John Connolly Division Director, APS Engineering Support Advanced Photon Source Accelerator Safety Workshop, SLAC, Aug 21-23, 2018 Recent Incidents and Lessons Learned at the APS

2. AGENDA • Brief Overview of the APS • Recent Incidents • APS Safety and Work Planning & Control (WPC) improvements • Continuous Improvement • Summary and Conclusion

3. Overview of the Advanced Photon Source… • APS at a glance for FY2017: • 7 GeV, 100 mA, 3rd Gen. hard energy synchrotron, commissioned 1996 • 68 operating beamlines; 5,000 hours • > 23,000 visits by > 5,700 unique users • > 6,400 experiments • > 1,200 protein structure deposits • > 21,000 total to date by APS, then ESRF with > 13,000, NSLS-II with > 8,000, SPring-8 with > 4,700) • > 180 (unique) industrial users in FY16 from pharma, energy, electronics, materials… • > 21,800 (as of Feb 1 2018) journal publications; • > 21% in high-impact journals (> 27,000 total publications)

4. …with APS Upgrade on the horizon

5. Recent incidents • Significant number of events at Argonne (~12) since May 2017 • Include events at the APS: Capacitor arc flash, student battery shock, and laser near miss • Resulted in a number of management actions • Lab-wide electrical safety stand-down • APS work pause • Lab-wide WPC improvements

6. Recent incidents Unexpected discharge of 15kJ capacitor • Lack of zero energy verification • The workers relied on a process intended to achieve zero energy state without verification as required • Employee concerns expressed at a prior-to-start work informal project review were not addressed. • Recommendation to identify full scope of work and overall project milestone was never completed. • The work proceeded without an adequate work plan or procedure. • Resulted in 2nd degree burns and ringing in the ears

7. Recent incidents Student Shock from 300 VDC Battery • Undergraduate student contacts both exposed leads on battery • Student working outside scope of work • Out of service equipment was being used • Resulted in a number of management actions • Lab-wide electrical safety stand-down • APS work pause – including user program • Sector permission to operate removed

8. Recent incidents Near Miss Eye Exposure, Class 4 IR Laser • Performing common maintenance task on fiber laser, involving visual inspection and cleaning • Felt pin prick sensation on finger – investigation revealed that laser had been left energized • Resulted in a number of management actions • APS stand down of all hazardous laser operations • Sector permission to operate removed, limited operations have been restored with compensatory actions • Incident investigation recently completed

9. Aps safety and WPC improvements • Lessons learned to date: • Communication about work must be clear and concise: • Across levels and between workers • Especially focused across interfaces • Expectations for carrying out work safely have to be clearly communicated, as well as what is required of staff to meet expectations e.g. training • Simple processes that invoke ISM principles work best • Safety must be a core value • Safety = quality; Safety = communication; Safety = trust

10. Specific Aps safety and WPC improvements Focused on clarity and accountability • Observation/conversations + SMART card reviews • Pre-job briefs • APS WPC procedure revision • WPC for Collaborative Access Teams (CATS) • High risk work reviews • High risk work register • High risk work authorizations • Argonne improvements: Electrical safety program, Improving How We Work

11. Observations + SMART card reviews across photon sciences • Observation/conversations in practice • Monthly completion tracked at directorate level • Effective “manage by walking around” tool • Expanded observation conversation interaction with Collaborative Access Teams (CAT) on APS beamlines by ALD and DALDs • Cross-group safety walk downs have begun to provide fresh perspectives on improvements • Additionally, pilot of SMART cards to provide more targeted observations – for AES division • Anyone in supervisory position required to submit 2 (or more) per month • Monthly and quarterly review for recognition, leading indicator(s)

12. Mandatory pre-job briefings • Planned interactive discussion occurring very close to task start • Involves all related personnel, team members, stakeholders • Opportunity to reinforce a number of ISM principles, as shown at right including: • Roles & responsibilities • Pause/suspend/stop work authority • Authorization to start work • Focused dialogue rather than a monologue • Include a post-job briefing in order to be closed out Work Request number (if applicable) Link to Argonne electrical pre-job brief form added Clear identification of WCD Link to permit required for RSS configuration control Authorization to start work Reinforcement of role/responsibility and stop work authority Unambiguous assignee and date Post-job briefing capture based on criteria above

13. Aps wpc procedure revision • Procedure returns to fundamentals of ISM • All APS work classified into 3 categories: • Experimental work • Contractor / construction work • Facility technical tasks (aka “other work”) • Procedure also contains clear, essential elements of: • Work Control Document (WCD), • Pre-job briefs at the point of work, • Pause/suspend/stop work authority + obligation to stop, • Clear identification of approval versus authorization, • WP&C roles and responsibilities by work function

14. Work planning and control for collaborative access teams (CATs) • Information and training sessions held May 21 and May 29 (make-up) • CAT members required to use the same WPC principles and tools as APS staff

15. High risk work reviews • In Dec 2017, started a monthly ALD review of high risk planned work • Attendance by ALD office, HSE representatives, division senior mgmt, APS Upgrade, and others (e.g. User Office) • Keeps all appraised of high risk, unique or one-off work activities planned, methods to mitigate risk, and status of ongoing activities • Additionally, PSC has instituted: • Floor Coordinator / QEW walk downs prior to APS runs • Floor Coordinator completion of ESAF review checklist for each experiment prior to authorization • New software tool for improved request and allocation of DEEIs

16. High RISK Work register • Created in March 2018 as a central report of APS moderate to high risk work • Allowed for clear identification of tasks with elevated risk levels, Person In Charge, authorizer and information exchange with DOE’s Argonne Site Office. • Argonne tool rolled out later took advantage of PSC application experience

17. High risk work authorizations • High risk work requires Division Director authorization • Face-to-face review held with Person In Charge, SMEs, responsible Group Leader, ESH Coordinators, division management, others to review: • Documentation, readiness and authorization • WCDs, work site walk-downs, permits, PIC identified, authorization authority clear • Reviews and conversations • SMEs, peer reviews, design reviews, experimental safety reviews, other Lab reviews, lessons learned, additions to pre-job brief • If successful, results in memo to file (example at right) demonstrating clear authorization to proceed for high risk work

18. PSC involvement in Argonne improvements • Electrical safety program improvements • Improving How We Work project focused on WPC improvements in: • Documentation and procedures: WPC manual (like Electrical Safety Manual), tiered WCDs such as “WCD-EZ” • Training and support: Minimize CBT, enhance skill-based training, create “field assistance team”, streamline WPC application • Communication: Focused conversations on high risk work; leadership reinforcing examples of positive WPC practices; SAFE training • PSC well represented in all of the focus areas above.

19. continuous improvement Much accomplished, but self-critique identifies opportunities for improvement • Procedures alone don’t create change • Reinforce ISM / WPC principles until it becomes the way we work • Clear authorization for complex activities • Work at the APS can involve workgroups across divisions, workgroups within Argonne, contractors, subcontractors, APS Upgrade on a single effort • Optimize method of authorization • Use of the Work Authorization & Execution (WAE) system piloted to date, rather than reliance on manual authorization  • Monitoring adoption and application of WPC by CATS

20. Blower sparking in Cabinet • Example of the sequence of actions going right during an event on May 10, 2018: • AES/Controls engineer responds to an communications issue for a Ethernet to serial converter (Moxa device). Converter is non-responsive and located in Bldg 420, RF5 Cabinet 7 in an extraction controls cavity rack. • Investigation noted that 3 devices – the Moxa device, an Acopian 5V power supply and a Hammond rack mounted blower, all plugged into a cabinet-mounted vertical power strip – were not receiving power. • Engineer noted next that the associated circuit breaker on the wall across from the RF cabinet, was tripped. • Engineer, a QEW 1, paused to discuss what safety protocol would apply in resetting the circuit breaker, with an experienced RF technician. • After reviewing what PPE was required, the circuit breaker was flipped, a popping sound was heard and breaker tripped again. • All 3 devices were unplugged from the power strip and breaker reset to determine power strip was receiving power. Plugged in Moxa device and Acopian power supply without issue. • Discussed plugging in blower while power strip was energized and determined it was safer to de-energize the power strip, plug in the blower, then re-energize the power strip via breaker reset. • To the engineer’s surprise, the breaker did not trip. In dressing up the blower power cables inside the cabinet to close the door, a pop was heard and sparks flew out of the blower louvers. • Blower was immediately unplugged, removed from the cabinet, taken to a lab for investigation. The short of the power cord wiring to a capacitor housing was evident as shown at right and engineer stopped work to notify management. • AES DD, ADD, CTRLS Group Leader, PSC Safety Manager, and CTRLS Engineer walked down area after brief discussion in the CTRLS lab where blower was taken. • Work had valid WCD and pre-job brief held. Blower motor is UL recognized, CSA certified. Whole blower assembly is not NRTL listed but was DEEI inspected. • Extent of condition follow up: This blower was a new design recently installed. Similar blowers removed from service, vendor contacted. Above: Hammond Manufacturing Rack Mount Intake Blowers HB Series, installed in RF5 cabinet 7 Below: Burn marks from bare wire contact, sparking and short, tripping local circuit breaker on nearby panel

21. Summary and Conclusion • Number of Significant Events in the Last Year • Response to Lessons Learned • Improvements in WPC and ANL programs • Continuous Improvement • It Can Go Right

22. Questions? www.anl.gov