IRRS REFRESHER TRAINING Lecture 3

1. IRRS REFRESHER TRAININGLecture 3 Evaluation and Documentation of Observations

2. Outline • Learning Objectives • Definitions • Formulation of Observations • Experience from Past Missions Refreshing IRRS Training - Evaluation of Observations

3. Learning objectives After finishing this module the trainee will: be aware of the observation types to be offered in IRRS missions; be able to select the observation-type adequate to the review findings; know the process of formulating observations in a mission; have some idea about the statistical properties of the observations. Refreshing IRRS Training - Evaluation of Observations

4. definitions Refreshing IRRS Training - Evaluation of Observations

5. Observations and their types Observation:focused results of the review process, based on facts and related to possible ways of improvement or achievements to recognise, with reference to requirements in IAEA Safety Standards Findings In brief: observations • Observations lead to: • Recommendation • Suggestion • Good Practice • (Remarkable Performance) Refreshing IRRS Training - Evaluation of Observations

6. Recommendation (1) • Recommendations are proposed where aspects relative to the IAEA Safety Requirements are missing, incomplete, or inadequately implemented. • Recommendationsshould be • Specific • Realistic • Designed to result in tangible improvements to regulatory effectiveness (within the individual context of the Host Country) Refreshing IRRS Training - Evaluation of Observations

7. Recommendation (2) • Recommendationsshould also • be based on IAEA Safety Requirements, with the basis documented (Safety Guide may not be a basis); • be formulated in such a way that they are succinct and self-explanatory; • clearly specify the responsible party; • use “should”-language. Refreshing IRRS Training - Evaluation of Observations

8. Recommendation (3) When a recommendation is formulated it should • be based on checked and evidenced factual information • advise on what improvement in the regulatory regime should be made; • not advise on how it should be completed; • address the root causes, rather than the symptoms Refreshing IRRS Training - Evaluation of Observations

9. Suggestion (1) Suggestions are proposed whenopportunities for improvement not directly related to inadequate conformance with IAEA Safety Requirementsare identified. Suggestions maycontribute to improvements in national regulatory arrangements but are primarily intended to make the regulatory body’s performance more effective or efficient Refreshing IRRS Training - Evaluation of Observations

10. Suggestion (2) Suggestions • should stimulate the RB’s management and staff to consider new approaches; • may be proposed in conjunction with a recommendation; • should be based either on IAEA Safety Requirements or Safety Guides, with the basis documented; • formulated in such a way that they are succinct and self-explanatory; • clearly specify the responsible party; • use “should consider”-language. Refreshing IRRS Training - Evaluation of Observations

11. Good Practice (1) • A good practice is identified in recognition of an outstanding organization, arrangement, programme or performance superior to those generally observed elsewhere • A good practice • goes beyond the fulfilment of current requirements or expectations; • is worthy of the attention of other regulatory bodies as a model • might be applied elsewhere; • shall reference a basis in IAEA Safety Requirements or Safety Guides; • shall be clearly documented in the mission report. Refreshing IRRS Training - Evaluation of Observations

12. Good Practice (2) Thus a good practice should have the following characteristics: • Has proven benefit; • Be replicable and applicable elsewhere; • Be effective and creative; • Produce good results; • Does not contradicts a finding; • Be novel or unique. Refreshing IRRS Training - Evaluation of Observations

13. Formulation of observations Refreshing IRRS Training - Evaluation of Observations

14. Format of an Observation Quotation from the relevant IAEA Safety Standard that is to be / has been followed. (There may be more than one Bases) Recommendation / suggestion / good practice offered by the reviewer A description of what the reviewer observed, that has lead to the present finding or good practice Basic IRRS Training - Evaluation of Observations

15. Recording Observations During interviews and direct observations, reviewers should make detailed notes of all relevant information gained. On the techniques and content of taking notes see Training Module No. 5 (Methodology of the review) Note here too that the number of observations in an IRRS mission does not represent a measure of the current performance of the reviewed RB relative to any other RB. Refreshing IRRS Training - Evaluation of Observations

16. Formulation of Observations The reviewer is expected to formulatein writing his/her observations daily Observations of the day need to be presented on the Daily Team Meeting. Be aware that the first set of findings (R+S) needs to be ready as early as on the third day of the mission!! (For more details see Training Module No. 8 on Scheduling IRRS Missions) Good practices may be formulated later Refreshing IRRS Training - Evaluation of Observations

17. Finalization of an Observation Each observation is discussed and agreed upon by the team in Daily Team Meetings Agreed observations are finalized and inserted into the mission report by the reviewer who formulated them Observations agreed by the team should be discussed by the reviewers with the host counterparts for ensuring factual correctness of the observation No agreement on the findings with the host counterpart is needed or sought! Refreshing IRRS Training - Evaluation of Observations

18. Use of Action Plans Hosts provide Action Plan(s) based on their Self-Assessment Actions in the AP do not imply that they may not be IRRS findings (think of the follow-up m.) Please consult the AP’s for input! Refreshing IRRS Training - Evaluation of Observations

19. Experts say Refreshing IRRS Training - Evaluation of Observations

20. Recommendation of Experienced ReviewersGive particular attention to: Refreshing IRRS Training - Evaluation of Observations R,S & GP should be adequately justified in the text The text should be based on fact and evidence (not assumptions) and the conclusions justified against IAEA’s Standards. The text should be succinct and self-explanatory – not an exhaustive explanation but should include some examples. The focus for balance, harmonization and consistency of R, S & GPs should reflect the main purpose of the mission which is to improve the regulation of nuclear safety for the Host Country and the International Regulatory Community Some of the observations may fit in principle with the concept of Suggestions but are considered by the team to be beneath the threshold for inclusion as formal Suggestions – these should be considered for inclusion as observations in the main text only In the formulation of the mission findings the team leaders should consider the application of the principle of graded approach

21. Experience form past missions Refreshing IRRS Training - Evaluation of Observations

22. Collections of Observations Refreshing IRRS Training - Evaluation of Observations

23. Statistics of Observations (1) All missions Without outliers Follow-up missions Refreshing IRRS Training - Evaluation of Observations

24. Statistics of Observations (2) Follow-ups • R + S is negatively correlated with GP • In most missions the no. of findings is in the range of 40 - 60 Refreshing IRRS Training - Evaluation of Observations

25. Statistics of Observations(3) • The number of recommendations and that of suggestions are strongly positively correlated The number of recommendations and that of good practices are strongly negatively correlated Refreshing IRRS Training - Evaluation of Observations

26. Performance Indicators (1) Balance of findings: • Many Rs implies few GPs and vice versa  NR*NGP  const • The No of Rs and that of Ss are nearly proportional  NR/NS  const Refreshing IRRS Training - Evaluation of Observations

27. Performance Indicators (2) Action Plan Coverage: No. of findings also in the AP No. of findings not in the AP Refreshing IRRS Training - Evaluation of Observations

28. Warning Note once more: The number of observations have certain definite statistical properties, as such their relative values may have orientating merits within a given mission, yet their absolute values should not be used for any comparison of separate missions. Refreshing IRRS Training - Evaluation of Observations

29. Summary (1) • Three types of observations have been defined: • Recommendation • Suggestion • Good Practice • Observations need to refer to IAEA Safety Standards (Requirements or Guides) • The various observations have specific features, yet all need to be based on factual information and not on assumptions • Inputs: ARM (including SA), Action Plans, interviews, direct obervations Refreshing IRRS Training - Evaluation of Observations

30. Summary (2) Reviewers have to take notes and formulate their observations daily The first set of observations is expected to be ready at the end of the third day of the mission Observations are inserted into the mission report in a standard table format The number of recommendations, suggestions and good practices follow certain correlated tendencies. Yet comparisonof such numbers between IRRS missions to various countries should not be attempted Refreshing IRRS Training - Evaluation of Observations

31. THANK YOU FOR YOUR ATTENTION This activity is conducted by the IAEA, with funding by the European Union. The views expressed in this presentation do not necessarily reflect the views of the European Commission Photo: V. Friedrich Refreshing IRRS Training - Evaluation of Observations

32. Remarkable Performance* * Under consideration, not yet official • Remarkable performance is identified in recognition of an organization, arrangement, programme or performance that – although is not unique and thus does not qualify for a good practice – is substantially superior to the general international practice • Remarkable performance • is meant to appreciate high level performance or practice of the RB; • in all other attributes is similar to good practices. Refreshing IRRS Training - Evaluation of Observations