Monitoring and Evaluation Catherine Connor Ben Russill David Tye Harvey Hurree

1. Monitoring and Evaluation • Catherine Connor • Ben Russill • David Tye • Harvey Hurree

2. Agenda

3. The Quality Code (Chapter B8) sets out the following Expectation about programme monitoring and review, which higher education institutions are required to meet. Higher education providers have effective procedures in place to routinely monitor and periodically review programmes. The following Expectation is also relevant to this topic: Higher education providers have in place effective processes to approve and periodically review the validity and relevance of programmes (Chapter A4: Approval and review).

4. The IndicatorsIndicator 1Institutions ensure that their responsibilities for standards and quality are dischargedeffectively through their procedures for the monitoring and review of programmes.Indicator 2Institutions ensure that the overriding responsibility of the academic authority (for example senate or academic board) to set, maintain and assure standards is respected and that any delegation of power by the academic authority to review programmes is properly defined and exercised.Indicator 3Institutions make use of external participation at key stages for the review of programmes, as independence and objectivity are essential to provide confidence that the standards and quality of the programmes are appropriate.

5. Indicator 4Monitoring and review processes are clearly described and communicated to those who are involved in them. Indicator 5Institutions routinely monitor (in an agreed cycle) the effectiveness of their programmes:• to ensure that programmes remain current and valid in light of developing knowledge in the discipline, and practice in its application• to evaluate the extent to which the intended learning outcomes are being attained by students• to evaluate the continuing effectiveness of the curriculum and of assessment in relation to the intended learning outcomes• to ensure that recommendations for appropriate actions are followed up to remedy any identified shortcomings.

6. Indicator 6 Institutions periodically undertake a broader review of the continuing validity and relevance of programmes offered. Indicator 7 In the event of a decision to discontinue a programme, measures are taken to notify and protect the interests of students registered for, or accepted for admission to, the programme. Indicator 8 Institutions have a means of assessing the effectiveness of their programme monitoring and review practices.

7. Course design and approval linked to the Framework for Higher Education Qualifications • Course design and approval • Appointment of External Examiner • Course and module monitoring Quality Standards • Moderation of coursework and examination assessment by External Examiner • Course and module modification • Student feedback/Student engagement • First and Second internal marking • Course Review • Sample moderation of assessment (coursework and examination scripts)

8. Quality Cycle at London Met

9. Course design and approval linked to the Framework for Higher Education Qualifications • Course design and approval • Appointment of External Examiner • Course and module monitoring Quality Standards • Moderation of coursework and examination assessment by External Examiner • Course and module modification • Student feedback/Student engagement • First and Second internal marking • Course Review • Sample moderation of assessment (coursework and examination scripts)

10. Monitoring & Evaluation Working Group (MEWG) Set up April 2013 Chaired by Dominic Palmer Brown All 4 faculties represented Office of Institutional Effectiveness Academic Registry QEU

11. Monitoring & Evaluation Working Group (MEWG) • Tasks • To consider the effectiveness and value of current monitoring and evaluation procedures • To consider the UK Quality Code Expectations in respect of monitoring and evaluation • To consider sector good practice in respect of monitoring and evaluation • Objectives • To recommend a cohesive, timely and effective set of quality monitoring and evaluation practices to the University’s Undergraduate and Postgraduate Committee • To identify actions (quick wins) that might be achieved quickly and easily • To identify resources that might yield early results or that could deliver impact

12. Monitoring & Evaluation Working Group (MEWG) What did it look at? Procedures • Module Logs • Module evaluation questionnaires • External Examiner reports • PEMS • Course Committees • Course Logs • PEMs • FAMs • Student Engagement • Employer feedback Data • RPA data • DLHE data • NSS data

13. Recommendations • New Module and Course Logs with prepopulated data (course logs only). • Move from consideration of module performance at an early July PEM to consideration of course performance and modules of concern, at a late July PEM. • PEM Chair Report (PCR) of Course and Module logs when final data are available in November. • Development of a newFaculty Annual Monitoring Statement (FAMS) that will provide the University with: • a) an evaluation of Faculty performance at undergraduate and postgraduate level, • b) confirmation of the implementation of the University’s quality assurance procedures, • c) information on validation/review/modification/deletion activity and d) an update on enhancement within the Faculty (QEU0022-FAMS). • New Course and Module Modification Deadlines – 2nd February 2015 for modifications that impact on programme planning and two weeks post PEM for modifications that arise from PEM discussions • Online course evaluation – pilot in May.

14. Monitoring and evaluation process Information and source Scrutiny Forum Core documentation Module monitoring data - available from OIE Annual Monitoring Livelink site Module Log Completed by Module Leader Supplementary module performance data - Published by Academic Registry Performance Enhancement Meeting(PEM) in July, then PEM Chair Report in November Marksheets - Component marks produced by Module Leader Course Log Data sections populated by OIE, then Commentary sections completed by Course Leader External Examiner comments - Module Confirmation Proforma (MCP) - Published by Academic Registry SSB Secretary Resultsfrom module questionnaires- fromFaculty records External Examiner Annual Report OR External Examiner written commentary in advance of PEM (where the External Examiner cannot attend the PEM) -available from the EE and QEU Livelink sites -Published by Academic Registry/Assessment on EE Livelink site Faculty UGPG Committee PEM Chair Report Data sections populated by OIE, then Commentary sections completed by PEM Chair Course (& SSB) monitoring data - Available from OIE Annual Monitoring Livelink site Course Committee minutes - from Faculty records Results from course questionnaires - from Faculty records NSS (previous year data only for July) Supplied by OIE via Faculty senior managers University UGPG Committee Faculty Annual Monitoring Statement (FAMS) Completed by Associate Dean DLHE data - Supplied by OIE in course monitoring data PTES data - Available from Research Office Boxes on the left denote information type and the documentation into which it feeds. Information is either supplied by the Office of Institutional Effectiveness (OIE) or must be actively sourced by the person responsible for the production of the core document. All Course Logs, PEM Chairs’ Reports, FAMS, and PEM minutes need to be provided to QEU who will hold them on Livelink.

15. Annual Monitoring for Collaborative Partners Information Documentation Scrutiny Module Level External Examiner Comment Module Logs are shared with the Academic Liaison Tutor Module Level Student Feedback Collaborative Module Log produced by collaborative partner Module Level Performance Data Course Level External Examiner Comment (aggregated from modules) Collaborative Course Log produced by collaborative partner Course Level Student Feedback (aggregated from modules) Collaborative Partner PEMs Course Level Performance Data (available from the Office of Institutional Effectiveness (OIE) the Academic Registry, and/or the collaborative partner) Collaborative Annual Monitoring Statement (CAMS) produced by collaborative partner Other relevant course information e.g. employment statistics (available locally) Faculty UGPG Committee Overall summary of good practice/issues to be addressed and annual update on partner activity Partnership Annual Monitoring Statement (PAMS) produced by Faculty Faculty update on partner activity University UGPG Committee QEU Collaborative Performance Report University evaluation of collaborative partner performance The boxes on the left denote the action and the documentation into which it feeds. Advice is supplied (variously) by the Quality Enhancement Unit (QEU), the Registry and the Office of Institutional Effectiveness (OIE). The boxes on the right denote the forum at which the annual monitoring documents are considered

16. Core Documentation for Module, Course, Subject and Faculty Monitoring & Evaluation Course Log Data sections populated by OIE, Commentary sections completed by Course Leader PEM Chair Report (PCR) Aggregated and individual data provided by OIE, Commentary sections completed by PEM Chair Faculty Annual Monitoring Statement (FAMS) Completed by AssociateDean Module Log Completed by Module Leader

17. Review Forums for Module, Course, Subject and Faculty Monitoring & Evaluation Performance Enhancement Meeting(PEM) PEM Chair’s Report (PCR) Faculty UGPG Committee University UGPG Committee July PEM December/ January University UGPG November PEM (Chair only) November Faculty UGPG

18. Thurs 3 July

22. Any module with a Pass rate of lower then 80% after the resit period is to be flagged as a module of concern and should be addressed in the November PEM Chair Report Any module with a Pass rate of lower then 60% is to be flagged as a module of concern. Data provided to faculty participants and managers to aid their preparation for the meting by the OIE November PEM The PEM Chair report takes into account relevant course logs when full year data is available July PEM Takes place as soon as feasible in the latter half of July Faculties produce FAMS using the PEM Chair Report . QEU0023a/b PG/UG module log and QEU0024a/b PG/UG Course Log Faculty FAMS considered at Faculty UG/PG committees Subject Standards Examiners Supplementary Module information provided by the Academic Registry Faculty FAMS considered at University’s UG/PG committee

23. PHASE 1: End June to beginning August Module results, Progression decisions, Awards, Other SRS data OIE PEM Level Data & NSS etc. External Examiner Input PEM Deliberations Selected Modules All Courses PEM Level Actions with owners confirmed by Chair OIE Data Draft Module Logs PEM Minutes Finalised Phase 1 Module Logs Draft Course Logs Finalised Phase 1 Course Logs Other Information Draft PEM Chair’s Report (PCR) (optional)

24. PHASE 2: End October to mid December Module results, Progression decisions, Awards, Other SRS data Updated OIE PEM Level Data & NSS etc. External Examiner Annual Reports PEM Chair Deliberations PEM Chair deliberations Modules Courses PEM Level Actions with owners confirmed by Chair Updated OIE Data Updated Module Logs (#1) Finalised Phase 2 Module Logs Finalised Phase 2 Course Logs Updated Course Logs (#2) Other Information FAMS Faculty UG/PG Ctee PEM Chair’s Report (PCR) Key: #1: New Module Logs required for PG dissertations and other summer modules, as results not available at Phase 1 #2: New Course Logs required for all PG, as awards and completion not available at Phase 1

26. Student Evaluation

27. Student Evaluation Pilot