Building an External Quality Assurance & Improvement Program

1. Building an External Quality Assurance & Improvement Program Brian Kruk | CIA, CISA, CGAP, CCSA, CCA Senior Director Quality Assessment Services

2. Agenda • A brief history of QA • Discuss the available QA&IP guidance • Examine common misconceptions in QA&IP development • Explore the differences between basic internal audit processes and effective components of a QA&IP • Utilization of the Old IIA PA 1311-2 to create an appropriate, right-sized QA&IP • Understand how a CMM can be used to facilitate the path to quality

3. Today’s Focus • Has anyone recently completed a QA? • Has anyone performed as a validator? • Is anyone working on their Internal Assessment or Self Assessment? • What do you want out of today’s session? • Are there any questions before we begin?

4. “ Quality is not an act – it is a habit.” - Aristotle “ Quality means doing it right whenno one is looking.” -Henry Ford

5. Quality Assessment • The process of evaluating the efficiency and effectiveness of an internal auditing organization through a comprehensive, qualitative review of audit procedures,leading to recommendations for improving controls, reducing risk and the introductions of successful innovative best practices. • It should also ensure compliance with the International Standardsfor the Professional Practice of Internal Auditing and other relevant organizational and departmental policies and procedures.

6. Synopsis of QA History • IIA first publication on QA in 1984 • IIA recommended peer reviews in previous Standards • IIA began conducting QAs in 1986 • Some QAs also conducted by other providers • GTF Brings Focus to Quality Initiative • QA Manual, 4th Edition, released in 2002 • QA Manual, 5th Edition, released in 2006 • QA Manual, 6th Edition, released in 2009 • QA Manual, 7th Edition, released in 2013 • QA Manual, 8th Edition, released in 2017

7. QAR 1984

8. Historical Situation Analysis on Standards • Consulting vs.Assurance Services • E-Commerce/Technology • Independence vs.Objectivity • ControlSelf-Assessment • CorporateGovernance • RiskManagement • ComplianceRequirements Not Coveredby Standards Standards Coverage Inadequate Standards in Conflict with BestPractices Standards Outdated

9. A Vision for the Future Professional Practices for Internal Auditing • Report of GTF to IIA Board of Directors • Adopt new framework • Revise definition of IA • Update Code of Ethics and Standards • Establish oversight committee • Develop guidance to support the Standards

10. Continuous Improvement Highlights Examples of Shortfalls • Addressing the applicability of the Standards for specialty groups • Further clarification of assurance and consulting services • Knowledge of key IT risk, controls and technology-based audit techniques • Periodic internal and external QA and ongoing monitoring as part of QA&IP • Inclusion of overall opinion and/or conclusion where appropriate, in final communications

11. Professional Practices Framework - 2002 • The “Path to Quality” gets its formal start with the creation of: • 7 New Quality Standards • 5 Practice Advisories

12. Continuous Improvement Highlights • By Jan. 2004 – 24 changes to the PFF • 11 new Standards • 13 additions to glossary • 11 new practice advisories • 5 revisions to PA’s

13. Continuous Improvement Highlights • July 2007 – Arrival of the New International Professional Practice Framework

14. Continuous Improvement Highlights • By the end of 2009 – changes to the IPPF • 6 new Standards • 19 new interpretations • 13 additions to glossary • Practice advisories reduction to 58 • 3 new practice guides • New 13 GTAG’s • New 3 GAIT’s

15. Continuous Improvement Highlights • 2010 to 2011 – changes to the IPPF • 3 new, 1 deleted • 15 revised Standards • 9 new and revised interpretations • 5 revisions to glossary • 13 new practice advisories • 8 new practice guides • 3 new GTAG’s

16. QA Related Implementation Guides • IG1300 - Quality Assurance and Improvement Program • IG1310 - Requirement of the Quality Assurance and Improvement Program • IG1311 - Internal Assessments • IG1312 - External Assessments • IG1320 – Reporting on the Quality Assurance and Improvement Program • IG1321 - Use of “Conforms with the International Standards for the Professional Practice of Internal Auditing” • IG1322 – Disclosure of Nonconformance • PG – Measuring Internal Audit Effectiveness and Efficiency • PG – Quality Assurance & Improvement Program • Old PA 2120-2 Managing the Risk of the Internal Audit Activity

17. Structure of Implementation Guides • Getting Started • Considerations for Implementation • Specific Related Topics • Example: IG1311 – Internal Assessments • On-going Monitoring • Periodic Self-Assessment • Considerations for Demonstrating Conformance

18. Continuous Improvement Highlights

19. Attribute Standards • 1000: Purpose, Authority and Responsibility • 1100: Independence and Objectivity • 1200: Proficiency and Due Professional Care • 1300: Quality Assurance and Improvement Program

20. Performance Standards • 2000: Managing the Internal Audit Activity • 2100: Nature of Work • 2200: Engagement Planning • 2300: Performing the Engagement • 2400: Communicating Results • 2500: Monitoring Progress • 2600: Management’s Acceptance of Risks

21. QA Related Standards 1300 – Quality Assurance and Improvement Program (New) The chief audit executive must develop and maintain a quality assurance and improvement program that covers all aspects of the IAAand continuously monitors its effectiveness. This program includes periodic internal and external quality assessments and on-going monitoring. Each part of the program should be designed to help the IAA add value and improve the organization’s operations and to provide assurance that the IAA is in conformity with the Standards and the Code of Ethics. Interpretation: • A quality assurance and improvement program is designed to enable an evaluation of the internal audit activity’s conformance with the Definition of Internal Auditing and the Standards and an evaluation of whether internal auditors apply the Code of Ethics. The program also assesses the efficiency and effectiveness of the internal audit activity and identifies opportunities for improvement.

22. QA Related Standards Previous 1310: Quality Program Assessments • The internal audit activity should adopt a process to monitor and assess the overall effectiveness of the quality program. The process should include both internal and external assessments Current 1310: Requirements of the Quality Assurance and Improvement Program • The QA&IP must include both internal and external assessments.

23. QA Related Standards Current 1311 – Internal Assessments • Internal assessment must include: • Ongoing monitoring of the performance of the IAA. • Periodic reviews performed through self-assessment or by other persons within the organization with sufficient knowledge of internal audit practices.

24. QA Related Standards (New IPPF) Current 1311 – Internal Assessments • Interpretation: Ongoingmonitoring is an integral part of the day-to-day supervision, review and measurement of the IAA. Ongoing monitoringincorporated into the routine policies and practices used to manage the IAA and uses processes, tools and information considered necessary to evaluate conformance with the DIA, COE and Standards. • Periodic reviews are assessments conducted to evaluate conformance with the DIA, COE and Standards. • Sufficient knowledge of IA practices requires at least an understanding of all elements of the IPPF.

25. QA Related Standards Original 1312 – External Assessments • External assessments such as quality assurance reviews, should be conducted at least once every five years by a qualified, independent reviewer or review team from outside the organization.

26. QA Related Standards 1st Subsequent Revision1312 – External Assessments • External assessments should be conducted at least once every five yearsby a qualified, independent reviewer or review team from outside the organization. • The potential need for more frequent external assessments as well as the qualifications and independence of the external reviewer or review team, including any potential conflict of interest, should be discussed by the CAE with the board. • Such discussions should also consider the size, complexity and industry of the organization in relation to the experience of the reviewer or review team.

27. QA Related Standards Current 1312 – External Assessments • External assessments must be conducted at least once every five years by a qualified independent reviewer or review team from outside the organization. The CAE must discuss with the board: • Theneed for more frequent external assessments. • The qualifications and independence of the external reviewer or review team, including any potential conflict of interest.

28. QA Related Standards (New IPPF) 1312 – External Assessments Old Interpretation: A qualified reviewer or review teamdemonstrates competence in two areas: the professional practice of internal auditing and the external assessment process.Competencecan be demonstrated through a mixture of experience and theoretical learning.Experiencegained in organizations of similar size, complexity, sector or industry, and technical issues is more valuable than less relevant experience. In the case of a review team, not all members of the team need to have all the competencies;it is the team as a whole that is qualified. The chief audit executive uses professional judgment when assessing whether a reviewer or review team demonstrates sufficient competence to be qualified. An independent reviewer or review team means not having either a real or an apparent conflict of interest and not being a part of, or under the control of, the organization to which the IAA belongs.

29. QA Related Standards (New IPPF) Current 1312 – External Assessments Current Interpretation: External assessment may be accomplished through a full external assessment, or a self-assessment with independent external validation. The external assessor must conclude as to the conformance with the COE and the Standards; the external assessment may also include operational or strategic comments. A qualified assessor or assessment teamdemonstrates competence in two areas: the professional practice of internal auditing and the external assessment process.Competencecan be demonstrated through a mixture of experience and theoretical learning.Experiencegained in organizations of similar size, complexity, sector or industry, and technical issues is more valuable than less relevant experience. In the case of an assessment team, not all members of the team need to have all the competencies;it is the team as a whole that is qualified. The chief audit executive uses professional judgment when assessing whether a reviewer or review team demonstrates sufficient competence to be qualified. An independent assessor or assessment team means not having either an actual or perceived conflict of interest and not being a part of, or under the control of, the organization to which the IAA belongs. The CAE should encourage the board oversight in the external assessment to reduce perceived or potential conflicts of interest.

30. QA Related Standards Old 1320 – Reporting on the Quality Assurance and Improvement Program • The CAE mustcommunicate the results ofthe quality assurance and improvement program tosenior management andthe board. Current 1320 – Reporting on the Quality Assurance and Improvement Program • The CAE must communicate the results ofthe quality assurance and improvement program to senior management and the board. Disclosure should include: • The scope and frequency of both the internal and externa assessments • The qualifications and independence of the assessor(s) or assessment team, including potential COI • Conclusion of assessors • Corrective action plans

31. QA Related Standards (New IPPF) 1320 – Reporting on the Quality Assurance and Improvement Program • Current Interpretation: • The form, content and frequency of communicating the results of the QA&IP is established through discussions with the senior management and the board and considers the responsibilities of the IAA and CAE as contained in the IA Charter. To demonstrate conformance with the COE, and the Standards, the results of external and periodic internal assessments are communicated upon completion of such assessments and the results of ongoing monitoring are communicated at least annually. The results include the assessor’s or assessment team’s evaluation with respect to the degree of conformance.

32. QA Related Standards Old 1321 – Use of Conforms with the International Standards for the Professional Practice of Internal Auditing • The chief audit executive may state that the IIA conforms with the ISPPIA only if the results of the QA&IP supports this statement. Current 1321 – Use of Conforms with the International Standards for the Professional Practice of Internal Auditing • Indicating that the IAA conforms with the ISPPIA is appropriate only if supported by the results of the QA&IP.

33. QA Related Standards Previous 1322 - Disclosure of Nonconformance • Although the internal audit activity should achieve full compliance with the Standards and internal auditors with the Code of Ethics, there may be instances in which full compliance is not achieved. When noncompliance impacts the overall scope or operation of the internal audit activity, disclosure should be made to senior management and the board. Current 1322 - Disclosure of Nonconformance • When nonconformance with the COE, or the Standardsimpacts the overall scope or operation of the IIA, the CAE must disclose the nonconformance and the impact to senior management and the board.

34. Full External Assessments

35. External QAProvider Resources TheIIA IndustryGroups ConsultingFirms LocalPeers

36. External Assessments • Areas of focus • Review IA Activity’s charter, audit plans, policies and procedures • Review a sample of audit reports, special projects and supporting work papers • Review staff composition, supervision, professional development and response to client needs

37. External Assessments • Areas of focus • Assess staff and client satisfaction through interviews and surveys • Specifically interview audit committee chairperson, a representative sample of officers, senior executives and management clients and the external auditing partner • Risk assessment methodology • Approach and adequacy of IT audit coverage

38. External Assessment Activities • Tools review • Self study/benchmarking • Customer/staff survey • On-site activities • Interviews (board, management, external auditor, staff) • QA Program • Work paper reviews • Issue report

39. QA – Assessment Objectives • Assess the efficiency and effectiveness of the internal audit activity in light of: • Its charter and mission • Expectations of the board, senior management, audit clients, and the CAE • Identify opportunities and offer ideas and counsel to the CAE and staff for: • Improving their performance • Increasing the value they add to the enterprise • Provide an opinion on the internal audit activity’s conformance to the spirit and intent of the Standards

40. QA – Assessment Approach • Advanced prep and CAE questionnaire • Survey of clients and staff • Interviews with senior managers and staff • Review tools (programs) • Background • Governance • Staff • Management • Process • Information technology • Rating of conformity with IIA Standards

41. QA – Conforming Evaluation Definitions • GC – “Generally Conforms” means the assessor has concluded that the Activity’s charter, structure, policies, and procedures, as well as the processes by which they are applied, are judged to be in conformity with a majority of the Standardswith some opportunities for improvement being possible. • PC – “Partially Conforms” means the assessor has concluded that a good faith effort exists but deviations from conformity for a majority of the Standards exists and corrective action is needed. These deviations are not, however, significant enough to preclude the Activity from carrying out its responsibilities in an acceptable manner. • DNC – “Does Not Conform” means the evaluator has concluded that the Activity is not aware of, is not making good-faith efforts to comply with or is failing to achieve conformity with the majority of the Standards,thus impacting its ability to carry out its mission.

42. QA Overall Evaluation OVERALL EVALUATION Generally Conforms (GC) Attribute StandardsGC 1000 Purpose, Authority & Responsibility GC 1100 Independence & Objectivity GC 1200 Proficiency and Due Professional Care GC 1300 Quality Assurance and Improvement PC • Performance Standards GC 2000 Managing the IA ActivityGC 2100 Nature of Work GC 2200 Engagement Planning GC 2300 Performing the Engagement GC 2400 Communicating Results GC 2500 Monitoring Progress GC 2600 Communicating the Acceptance of Risk GC IIA Code of Ethics GC

43. QA – Potential Issues Reporting Categories • Opportunities to improve conformity with Standards • Opportunities for IA consideration • Suggestions for senior management • Verbal comments

44. Self-Assessment with Independent Validation

45. Self-assessment w/ IndependentValidation • Benefits vs.Shortcomings • Perceived as lesscostly • Perceived as lessintrusive • Can generates IA teambuy-in • Can be a training & process improvementexercise • Documentation process morecumbersome • Perceived as lessthorough • Less independent andobjectivity • Lessen opportunity for best practicecomparisons

46. Performing the Validation • Key points for consideration • General considerations • Planning and preparation • Interviews • Self-assessment fieldwork • Self-assessment results, recommendations and implementation plans

47. Performing the Validation Key Points for Consideration • Perception of lower cost – more time invested by IA Activity • Project timeline controlled by IA Activity • No or limited best practice enhancements • Less independent as much of the work is done by the IA Activity • Key Point – validator should be qualified • Interview and survey limitations

48. Performing the Validation Overview and Details • General considerations • Planning and preparation • Interviews • Self-assessment fieldwork • Self-assessment results, recommendations and implementation plans

49. Performing the Validation General Considerations • Alternative means for complying with Standard 1312 external assessments • Benefits • Economics/practicality • Expand external assessments to more IA activities

50. Performing the Validation General Considerations • Scope limitations • Scope more targeted/limited than full external assessment • Focused on basic IA expectations • Fulfillment of IA mission • Conformance to the Standards • Areas where in-depth analyses may be curtailed or excluded