Anne Kerhoas, IAEA

Regional Training Course on Transition to IMS Approach Supporting the Development of a Strong SCJiujiang City 8-12 November 2010 Anne Kerhoas, IAEA SCARTSafety Culture Assessment Review Team Approach, feedback and future plans

OUTLINES • SCART approach • SCART mission process • SCART missions carried out • Future plans

BACKGROUND • SCART is a safety review service, which reflects the expressed interest of Member States for methods and tools for safety culture assessment. • It is a replacement for the earlier service ASCOT(Assessment of Safety Culture In Organization Team)

SCART APPROACH • The SCART mission objectives • To provide a valid assessment of Safety Culture considering behaviours and attitudes; • To identify strengths and areas for improvement of Safety Culture; • To provide recommendations and suggestions for improvement, and To provide other Member States with information on good practices in Safety Culture

SCART review scope-functional areas SCART includes all major organizational functions, as formulated in Safety Guide NS-G-2.4: Operating Organization for Nuclear Power Plants: • Policy making functions • Operating functions • Supporting functions • Reviewing functions

Responsibility & Commitment CEO Directors Department Managers Section Managers Foremen STAFF SCART Review scope- responsibility levels • Members of Boards of Directors • Chief Nuclear Officers and Executive Officers • Station Directors and senior managers at nuclear facility level • Middle managers (operative nuclear facility level) • First line supervisors (team level, managing “face to face” groups) • Shop floor level (operators, maintenance staff, technicians, etc.)

SCART Methodology–assessment variables SCART is a Safety Review Service, which evaluates the main characteristics of safety culture in a nuclear facility with respect to IAEA Safety Standards The assessment variables are the attributes of the five safety culture characteristics identified in IAEA GS-G-3.1 in 2006 The attributes are formulated as positive descriptions of a specific performance aspect in a nuclear facility, which, if fulfilled, would characterize this performance aspect as belonging to a strong safety culture

SCART Review approach Because of the intangibility of safety culture and the diversity of experiences and review skills among different reviewers, a comprehensive safety culture review has to be a team exercise. It is the integration, not the individuals, that leads to expertise of the SCART team.

SCART Assessment tools The main tools for the individual opinion building in SCART are: • Interviews • Observations • Documentation analysis (in addition to the analysis of the Advanced Information Package)

Interview techniques The main objectives of using interviews are • To gain an opinion on the actual awareness about the nuclear hazard • To gain an opinion on safety-relevant social norms and attitudes of management and staff • To enquire and infer possible reasons for observed inconsistencies or contradictions between actual and formally expected behaviour (prescribed in processes and procedures)

Observation techniques Observations, which may be particularly sensitive to safety culture • Shift changeover • Safety committee meetings • Other management meetings, not obviously related to safety • Routine inter-departmental meetings • Pre-job briefing by supervisors • Post-job review

Documentation analysis • Documents produced by the national regulatory authority, or on its behalf, such as: • Legislation, regulations or regulatory guides on human factors, management systems and safety culture • Safety culture documents of the nuclear facility (or the operating organisation) such as: • Policy statement on safety/safety culture • Nuclear facility arrangements for safety/safety culture, including assignment of responsibilities • Resource allocation and qualification of personnel, who deals with safety/safety culture • Documentation of recruitment and training activities

Individual opinion building • The process of data gathering in SCART aims at developing an individual reviewer’s opinion for all assessment variables (safety culture attributes as in GS-G-3.1.) • The individual opinion on the attributes is summarized by every reviewer both in comments for each attribute as well as in quantified ratings • Higher ratings mean greater correspondence between the attribute and the empirical data, or - formulated differently - greater evidence for a strong safety culture concerning the specific attribute

Interview data sheet (1/3)

Individual evaluation result spread sheet

Team opinion building • After the individual opinion building is accomplished, an integrated team opinion is achieved through: • Calculating the central tendency for the team (median is used for calculating the central tendency of the ratings of the reviewers) • Astutely managed team discussion • The combination of the two features is essential to ensure the validity of the SCART results • The scoring results of the team are used only within the team discussion, and are not part of the Mission report

Identifying strengths and areas for improvement • Strength Attributes, identified as strength, reflect a specific organizational performance or attitude at the nuclear facility which would characterize it as belonging to a strong safety culture. This is reflected in high SCART team ratings (rating values from 7 to 9). While not as unique or outstanding organizational or group feature, as good safety culture practice, identified strengths are worth serving as an example for the continuous improvement process of other aspects of safety culture at the nuclear facility

Identifying strengths and areas for improvement • Area for improvement Similarly to strengths, a safety culture attribute may be identified as an area for improvement. In areas identified for improvement, the organizational performance or attitude at the nuclear facility does not correspond to what is expected according to IAEA Safety Standards. Low SCART team ratings (rating values from 1 to 3) reflect areas for improvement. Identification of areas for improvement usually presupposes recommendations concerning the area

Formulating recommendations • Recommendation A recommendation is advice on how improvements in safety culture can be made in areas that have been identified as areas for improvement (scoring values 1 to 3). All the attributes rated 1 to 3 will be considered for one or several recommendations Recommendations should be based on IAEA Safety Standards and international experience and should address the identified deeper problems rather than the symptoms of the raised concerns

Formulating suggestions • Suggestion For those attributes, which are identified neither as areas for improvement nor as strengths (scoring values 4 to 6), it may appear necessary to formulate suggestions A suggestion is advice on how to continue on the way of improving safety culture, although no overt weaknesses have been identified. It is based on IAEA Safety Standards and international experience and is primarily intended to indicate enhancements of existing safety-related activities or approaches and to point out superior alternatives to these current activities or approaches

Identifying good safety culture practice ·Good practice A good practice is an indication of an outstanding organizational or group performance, related to safety culture, superior to performance observed elsewhere, and more than just the fulfilment of current requirements and expectations. It must be superior enough to be worth bringing to the attention of other nuclear facility operators as a model in the general drive for human performance excellence

SCART MISSION PROCESS Member State/ Nuclear Facility Formal agreement STEP 1 Initiation of the SCART mission Advanced Information Package AIP Preparation of the SCART mission STEP 2 Preparatory meeting Team composition Self-assessment Data gathering and opinion building Conduct of the SCART mission STEP 3 Mission schedule Preliminary report Team training STEP 4 SCART follow-up

Step 2-advanced information package (AIP) one example • List of NPP abbreviations and acronyms • NPP general information • NPP organization charts • List and brief description of NPP committees • List of counterparts for the SCART mission • Objectives and previsions for plant operation • Integrated Management System • Safety Culture Programme • Plan to improve safety culture and work quality • Fitness for duty programme • List of general procedures • List and brief description of Organization and Human Factors projects • Self-assessment approach and results • Safety culture survey approach and results

Step 3 – general schedule

Step 3 - interviews/general data As an example, the SCART team conducted in the Santa Maria de Garona NPP: • About 90 interviews • Of employees and some contractors • From most functional areas and all management levels • And 19 observations

Step 3 - data gathering and opinion building • One characteristic per day • 4 interviews per day • Fill in interview data sheet • 2 daily meetings For each characteristic, representatives of all functional areas as well as all responsibility levels are interviewed. Senior managers are interviewed at the end of whole process

SCART MISSIONS CARRIED OUT

Trends on findings- characteristics level Categories of findings A: Safety is a clearly recognized value B: Leadership for safety is clear C: Accountability for safety is clear D: Safety is integrated into all activities E: Safety is learning driven 30

Trends on recommendations and suggestions-characteristics level A B C D E A: Safety is a clearly recognized value B: Leadership for safety is clear C: Accountability for safety is clear D: Safety is integrated into all activities E: Safety is learning driven

FUTURE PLANS • One consultancy meeting(CS) took place in Vienna in June 2010 • Several suggestions have been made • One consultancy meeting in October 2010 to streamline the basis for evaluation • The SCART guidelines will be reviewed accordingly to the two CS meetings

List of actions suggested during the CS of June 2010 • Evaluation items based on the SC attributes from IAEA safety standard GS-G-3.1 shall be developed.(Action1) • A table reflecting through key words, the formal and non formal part of each evaluation item shall be developed. It shall also include the observables (Action2) • A table addressing the reference of IAEA safety standards for each evaluation item shall be developed (Action3)

List of actions (Cont.) • A mechanism shall be developed and implemented to enhance the interaction between outputs from observations and document review and the focus of interviews. (Action 4). • A table reflecting for each evaluation item the documents and records to be reviewed during the mission shall be developed. It shall be connected with the table of observables.(Action5)

List of actions (Cont.) • The rating system shall be reviewed and additional discussions shall take place. (Action 6) • The evaluation data sheet shall be reviewed addressing the following items: list of evidences gathered during interviews, observations, doc review, evaluation part composed of qualitative and rating part (Action 7) • In order to highlight the specificity of HOF, a guidance related to the wording shall be developed (Action8)

Conclusions and some lessons learnt • The two first missions at a nuclear facility allowed us to validate the main principles of the SCART methodology • The methodology worked well in the practice, and was accepted both by reviewers and counterparts • The combination of a safety specialist and a behavioural scientist in the team leadership is essential for the success of the Mission • The due consideration for language issues, like translation in interviews, and having at least one reviewer as native speaker appears to be important

Main conclusions for IAEA • SCART as a Safety Review Service for safety culture is a new level of support for Member States • Being a promotion tool for safety culture, it is a motor of motivation for the Member States • SCART missions will increasingly become opportunities to facilitate the application of IAEA Safety Standards

Anne Kerhoas, IAEA