hep quality audit manual draft v1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
HEP Quality Audit Manual (draft v1) PowerPoint Presentation
Download Presentation
HEP Quality Audit Manual (draft v1)

play fullscreen
1 / 93

HEP Quality Audit Manual (draft v1)

627 Views Download Presentation
Download Presentation

HEP Quality Audit Manual (draft v1)

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. HEP Quality Audit Manual (draft v1) © 2007 Martin Carroll, Salim Razvi, Tess Goodliffe Training Module 21 v1 Oman Accreditation Council

  2. Module Objectives Participants will: • Have a detailed knowledge of the HEP Quality Audit Manual draft v1; • Be able to prepare for the Quality Audit with confidence; • Develop a deeper appreciation of the Quality Audit process; • Develop a deeper appreciation of the nature of evidence in Quality Audit; • Have an opportunity to provide comments on the draft manual.

  3. The Manual’s Table of Contents Part A: Quality Audit Overview 9.00am – 9.30am Presentation, Q&A Part B: The Self Study 9.30am – 9.40am Presentation Part C: The External Review 9.40am – 10.00am Presentation 10.00am – 10.20am Short Break 10.20am – 10.40am Working Groups 10.40am – 11.00am Group Feedback Part D: Methods Of Analysis 11.00am – 11.20am Presentation 11.20am – 11.40am Working Groups 11.40am – 12.00pm Group Feedback Part E: Appendices

  4. PART A:QUALITY AUDIT OVERVIEW

  5. Provider QA Processes First cycle commencing HEP Accreditation 2008 HEP Licensure Stage 1 : Quality Audit 4 years ≤4 years Appeal HEP Accreditation HEP Accreditation Pass Certificate Stage 2 : Standards Assessment Fail KEY 1-2 years on Probation Start/End Pass Process HEP Accreditation Terminated HEP Standards Fail Reassessment Document

  6. Provider QA Processes HEP Accreditation Stage 1 : Quality Audit 4 years ≤4 years HEP Accreditation HEP Accreditation Pass Certificate Stage 2 : Standards Assessment STAGE 1: QUALITY AUDIT Self Study External Review External Quality Audit Quality Audit Self Study Review Portfolio Report Activities Activities

  7. Provider QA Processes HEP Accreditation Stage 1 : Quality Audit 4 years ≤4 years HEP Accreditation HEP Accreditation Pass Certificate Stage 2 : Standards Assessment STAGE 2: STANDARDS ASSESSMENT Self Assessment External Assessment Self Assessment Activities Standards Assessment Report External Assessment Activities Assessment Application

  8. Program QA Processes

  9. What is Quality Audit?

  10. What is a Quality Audit? In the Sultanate of Oman, a Quality Audit is a public and systematic determination of whether: • A HEP’s goals and objectives are based on appropriate regulations, standards and benchmarks; • Its planned arrangements are suitable to achieve those goals (i.e. check the overall approach); • Its actual practice conforms to the planned arrangements (i.e. check the deployment); • The arrangements achieve the desired results; • The organisation is learning from a self-evaluation of its approach, deployment and results, and can demonstrate improvements.

  11. What is a Quality Audit? In the Sultanate of Oman, a Quality Audit is a public and systematic determination of whether: • A HEP’s goals and objectives are based on appropriate regulations, standards and benchmarks; • Its planned arrangements are suitable to achieve those goals (i.e. check the overall approach); • Its actual practice conforms to the planned arrangements (i.e. check the deployment); • The arrangements achieve the desired results; • The organisation is learning from a self-evaluation of its approach, deployment and results, and can demonstrate improvements.

  12. Reviewing Graduate Destinations ADRI Cycle of Quality Assurance Quality Audit ‘Double Loop’ Process IMPROVEMENT APPROACH RESULTS DEPLOYMENT Internal Review of the System (results in a Portfolio section with Strengths & OFI) External Review of the System (results in Audit Report section with Coms, Affs, Recs) • Evidence examples: • Internal program review reports • Input obtained from industry & professional bodies • Resulting action plans • Reports on progress against actionplans • Evidence examples: • HEP graduate destination targets • Industry needs analysis • Careers Service Plans • Curriculum demonstrating links to industry needs Topic: Graduate Destinations • Evidence examples: • Graduate destinationstatistics • Evidence of employersatisfaction (e.g. surveyresults) • Evidence examples: • Interviews with academic staff,careers service staff, recent graduatesand employers ADRI is a model of quality assurance used by agencies in many countries around the world (e.g. AUQA in Australia) It can be applied to any topic – in this case, the analysis of a HEP’s graduate destinations

  13. What a Quality Audit is Not • Secret (although some of the information and deliberations will be confidential). • An ‘accreditation’ in itself (it is part of the overall accreditation system). • A measurement of performance against each OAC HEP standard (like Standards Assessment), but audit does use those standards to help define the scope (i.e. chapter headings in the Portfolio). • A summative assessment (it does not result in a pass/fail or a grade or score). • A strategic review (it is focused on how well a HEP is doing, not what future direction it should head in).

  14. Evaluations / Audits Accreditations National Agencies for Accreditation & Quality Assurance in Europe, 2005 Rolf Heusser, Chairman of European Consortium for Accreditation (2006) Mutual Recognition of Accreditation Decisions, INQAAE Workshop, Den Haag, Nederlands.

  15. Scope of Audit(section 3.4 in the Quality Audit Manual)

  16. What does Quality Audit Consider? Comparison with old ROSQA NEW SCOPE PREVIOUS SCOPE • Governance & Management • Student Coursework • Student Research • Staff Research & Consultancy • Community Engagement • Academic Support Services • Students & Student Support Services • Staff & Staff Support Services • General Support Services & Facilities • Mission, Goals & Objectives • Governance & Administration • Learning & Teaching • Student Services • Learning Resources • Facilities & Equipment • Financial Planning & Management • Staffing & Employment Procedures • Research • Community Relationships

  17. PART B:THE SELF STUDY

  18. Self Study Principles (1 of 2) • A Self Study is a rigorous and comprehensive evaluation of the HEP and all its significant areas of activity • The Self Study should be championed by the very highest levels of the organisation. • The Self Study should involve many people. • The Self Study will take time. There is a large amount of information to be collected, analysed, interpreted and reported.

  19. Self Study Principles (2 of 2) • A Self Study is evaluative, not just descriptive. The idea is to find out not just what is happening, but how well it is happening. If we think we are doing something well, can we prove it? • A Self Study will require valid and reliable quantitative and qualitative information. • If done well, the Self Study will also have a value to the HEP independent from its purpose as the submission document for Quality Audit.

  20. What is a Quality Audit Portfolio? • Covers the entire institution. • Explains the HEP, its mission and vision, its special characteristics and its context. • Provides a concise summary of the HEP’s quality assurance systems for each function. • Provides a robust analysis of the effectiveness of those systems • Identifies strengths. • Identifies where improvements are needed.

  21. What does a Portfolio Look Like? • Maximum 30,000 words; 100 pages (including Appendices). • In English (and Arabic, if necessary/requested). • Hard copy and PDF form. • List of Supporting Documents and optional supporting Materials. • Final version should be professionally edited, type set, printed and published.

  22. Typical Portfolio Table of Contents Chairperson’s Introduction Overview of [name of HEP] The Self Study Method 1. Governance and Management 2. Student Coursework 3. Student Research 4. Staff Research and Consultancy 5. Community Engagement 6. Academic Support Services 7. Students and Student Support Services 8. Staff and Staff Support Services 9. General Support Services and Facilities Appendices Supporting Material Index Available Supporting Material Index

  23. Each Chapter of the Portfolio should conclude with ‘Areas of Strength’ and ‘Opportunities for Improvement’ (OFI) (Although not every issue will necessarily conclude with a strength or OFI)

  24. Areas of Strength • ‘Areas of Strength’ refer to areas in which the HEP can be justifiably proud • An Area of Strength is where the HEP can show effectiveness in achieving intended results • An Area of Strength highlights an example of potential good practice for the Audit Panel

  25. Areas of Strength Examples Area of Strength #1 Hogwarts College has an effective system for annual course review conducted by the Academic Board that incorporates student evaluations, teacher reflections, industry input and a review of scholarship in the field and that leads to course improvements. Area of Strength #2 Hogwarts College provides excellent pathways for graduates into foreign universities through the establishment of eight active student exchange and articulation agreements.

  26. Opportunities for Improvement • During the Self Study, the HEP identifies issues in need of particular attention • Identifying an OFI demonstrates the HEP’s understanding of the issue and its commitment to taking appropriate action • Therefore, if verified, may result in an ‘Affirmation’ rather than a ‘Recommendation’ in the Audit Report • The main advantages of identifying OFIs is to aid the HEP’s ongoing quality improvement activities

  27. OFI Examples Opportunity for Improvement #1 Hogwarts College needs to implement a systematic approach to analysing and acting upon the feedback it receives from its student surveys. Opportunity for Improvement #2 Hogwarts College needs to redesign its research funding scheme in order to achieve desired results.

  28. PART C:THE EXTERNAL REVIEW

  29. Quality Audit Protocols(section 9 in the Quality Audit Manual)

  30. Quality Audit Protocols • Conflicts of Interest • Undue Influence • The Chatham House Rule • Transparency vs Protectionism • Sensitive Information • Complaints about the HEP

  31. External Reviewers(section 11 in the Quality Audit Manual)

  32. Register of External Reviewers • The OAC is assembling a Register of External Reviewers. The OAC has sought nominations. All decisions are made by the OAC Board. • Includes local and international leaders in academia, higher education management, professional bodies and industry. • Membership lasts for 3yrs, and may be renewed. • The External Reviewers will receive thorough training and an Auditor Manual. • Only External Reviewers in the approved Register may be used for Quality Audits (and other types of external panels).

  33. Quality Audit Panel Membership • Membership will be assembled approximately as follows (subject to HEP consultation): • Two academicians (1 local; 1 international) • Two people with HEP management experience (1 local; 1 international) • One person from a relevant industry or profession • Each panel is supported by an Executive Officer from the OAC. • Each Panel may be accompanied by an Observer (who has no role in the process).

  34. The Audit Process(see Table 1 in the Quality Audit Manual)

  35. Key Roles in the Audit Process

  36. KEY Process Decision Document Terminator / Link

  37. The Audit Visit(section 16 in the Quality Audit Manual)

  38. About the Audit Visit • Typically gets all the attention, but it is ONLY ONE PART of the overall quality audit process. • A chance for the Audit Panel to test the accuracy and completeness of the Portfolio. • Audit panel members can interview people and review resources and facilities. • May be 2+1 or 3+1 days, depending on the size and complexity of the HEP. • Highly professional, but also conducted in a positive and friendly manner.

  39. Dean Heads of Department Academic Staff Students (UG) External Stakeholders Laboratories Council Members Administrative Heads Academic Staff Students (PG) Student Services Library & IT TIME DAY 1 DAY 2 DAY 3 0900-0945 Interview 1 Interview 7 Callback interviews(if required) 1000-1045 Interview 2 Interview 8 Panel deliberationsand report drafting 1100-1130 Panel Review Panel Review 1130-1215 Interview 3 Interview 9 1230-1345 Interview 4 (lunch) Interview 10 (lunch) 1400-1500 Panel Review Panel Review 1500-1545 Interview 5 Interview 11 1600-1715 Interview 6 (on site) Interview 12 (on site) Verbal Feedbackto HEP 1730-1830 Panel Review Panel Review Example of a typical 2+1 Audit Visit program. Precise details will vary for each Audit.

  40. Interview Session Logistics • Unless otherwise arranged: • Maximum 8 interviewees per session. • All people will be selected by the Panel. • People should meet with the Panel once only. • Managers & staff interviewed separately. • The sessions are confidential, in that the Panel may use the information received, but may not reveal the identity of the provider (Chatham). • HEPs are also expected to respect the confidentiality of the process and may not coerce interviewees to say certain things or to divulge what was said.

  41. Random Interviews • Support collection of independent data. • Carried out by Panel members and take place during the audit visit. • Panel members will only speak with staff and students. • Potential interviewees are not obliged to participate • Interviews will usually be conducted confidentially with individual people. • The type of questions that will be asked during the random interviews are in Appendices M and N in the Quality Audit Manual. (see section 16.2.4 of the Quality Audit Manual)

  42. Public Submissions • Important aspect of accountability – supports the independence of the Quality Audit. • The public are invited (through the HEP, the local press and the OAC website) to submit their views. • Anonymous submissions will not be accepted. • Personal grievances and vague allegations will not be considered by the Panel. • Submissions will be considered in the context of the overall Quality Audit. (see section 15 of the Quality Audit Manual)

  43. Preliminary Feedback • Takes place at the end of the Audit Visit. • Audit Panel meets with HEP CEO and up to seven other HEP representatives. • Panel gives verbal feedback to HEP based on the preliminary indication of the Panel’s findings. • The feedback presented to the HEP during this session is confidential and not open for discussion. • This is preliminary feedback only and notbinding on the OAC. (see section 16.2.8 of the Quality Audit Manual)

  44. The Audit Report(section 17 in the Quality Audit Manual)

  45. About the Audit Report • A text report of about 40 pages. • Based on an analysis of all the evidence. • Public (posted on OAC website). • Does not include a pass/fail or a grade. • Does include Recommendations, Affirmations and Commendations (but not every topic considered will result in a Rec, Aff or Com). • Will not include every issue that was touched on during the audit, but will attempt to provide a balanced view of the HEP.

  46. Audit Report Table of Contents • Overview of Audit Process • Executive Summary of Findings • Summary List of Commendations • Summary List of Affirmations • Summary List of Recommendations • Chapters (mirroring the HEP’s Portfolio, but OAC could modify if required) • Appendix A: Audit Panel Members • Appendix B: Acronyms and Terms

  47. Audit Report Drafts

  48. Commendations • The Panel is looking for evidence of good practice. • Where the Panel finds an example of an effective process resulting in positive results, this may lead to a Commendation. • Especially if it is unusual, recent, or something that the rest of the sector could benefit from knowing about.

  49. Commendation Examples Commendation #1 The Oman Accreditation Council commends Hogwarts College for developing a framework for performance indicators that is demonstrably aiding the management and planning processes. Commendation #2 The Oman Accreditation Council commends Hogwarts College for successfully implementing a peer mentoring for academic staff system that has resulted in improved student satisfaction.

  50. Affirmations • Sometimes the Panel will find an OFI that the HEP has already identified and fully understood through its self review process AND which the HEP has made a clear commitment to addressing. • While this is still an OFI, it is also evidence of a self review system that works, and therefore should not be treated like other Recommendations.