Portfolio Committee on Basic Education Report on the Quality Assurance of the NSC

1. 12 February 2013 Dr Mafu S Rakometsi Portfolio Committee on Basic EducationReport on the Quality Assurance of the NSC

2. WHAT CAN WE LEARN FROM THE NSC RESULTS? Introduction to context, principles, approaches and processes Dr Mafu Rakometsi - CEO of Umalusi

3. Regulatory Framework • Quality Assurance of Assessment • NQF Act Section 27 (h) • The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework.

4. Section17 of the GENFETQA Act • (5) The Council must, with the concurrence of the Director-General and • after consultation with the relevant assessment body or education • institution, approve the publication of the results of learners if the • Council is satisfied that the assessment body or education institution • has— • (i) conducted the assessment free from any irregularity that may • jeopardise the integrity of the assessment or its outcomes; • (ii) complied with the requirements prescribed by the Council for • conducting assessments; • (iii) applied the standards prescribed by the Council which a learner is • required to comply with in order to obtain a certificate; and • (iv) complied with every other condition determined by the Council.

5. Framework for QA of Learner Achievement • Based on established and existing practices in assessment for certification • Prescribed components of External assessment (examinations) and Site based/ internal / continuous assessment • Use of systems, processes, and procedures to evaluate, inspect, monitor and report on examination systems, processes and procedures of public and private assessment bodies.

6. Framework for Quality Assurance of Assessment • Evaluation and /or accreditation of assessment bodies • Periodic inspection of assessment systems • Ongoing monitoring of assessment systems • Quality assurance of external examinations through: • Moderation of examination question papers • Monitoring and moderation of SBA • Monitoring the conduct of examinations • Moderation of marking • Standardization of assessment outcomes

7. Approval for the release of Results • Approval is based on the following requirements: • The examinations are conducted compliant to the applicable policies regulating the conduct and administration of the examinations • At the time of approval, there is no serious irregularity which could undermine the credibility of the examinations.

8. Quality Assurance of the DBE 2012 National Senior Certificate Examination Vijayen Naidoo – Sen. Manager : Quality Assurance of Assessments

9. PURPOSE • To ensure that the question papers are of the required standard (standard captured in the NCS and SAG’s) • To ensure that the question papers are relatively: - fair - reliable - representative of an adequate sample of the curriculum - representative of relevant conceptual domains - representative of relevant levels of cognitive challenge • External moderators Moderation of question papers

10. Moderation of the question papers Approach • Question papers set by panel of examiners – DBE • Internally moderated by DBE • Externally moderated by Umalusi • Subsequent moderations and approval.

11. Moderation of Question papers Number of NSC 2012 question papers moderate

12. Moderation of the question papers Criteria • Technical Criteria • Internal moderation – quality of internal moderator reports. • Content coverage • Cognitive Demand • Marking Guidelines • Language and Bias • Adherence to Policies and guidelines • Predictability

13. Moderation of the question papers Findings: Areas of Good Practice • Percentage of question papers and memoranda approved after first and second moderation ( Nov 2012- 79,6% ; Mar 13 – 72,3%) • Simultaneous moderation of final and supplementary question papers. • Standard of Internal moderation

14. Moderation of the question papers Findings: Areas of Concern • Adherence to timeframes and impact on quality of setting and moderation. • Question papers requiring more than four moderations. • Role of the external moderator.

15. Internal assessment refers to any assessment conducted by the provider, the outcomes of which count towards the achievement of the qualification • panels of moderators / subject specialists • Moderation of internal /continuous assessment

16. Purpose of Umalusi’s verification: • To verify the rigour and appropriateness of the DBE moderation process – linked to DBE plans • Ascertain the degree to which assessment bodies/provinces are attempting to ensure standardisation across • Ascertain the standard and quality of the tasks • Determine the extent and quality of internal moderation and feedback. • Determine the reliability and validity of the assessment outcomes Moderation of internal assessment (cont.)

17. Moderation of Internal Assessment • Phase 1 (June/July 2012)- focus on assessment instruments (teacher files)

18. Moderation of Internal Assessment • Phase 1: Findings • Quality and Standard of Assessed tasks • Content Coverage – mostly appropriate • Cognitive demand and difficulty levels –standards variable mainly at lower cognitive levels, • Over reliance on previous question papers. • Projects and practical investigations- not cognitively balanced, more theory than practice. • Marking tools – not applied consistently • Teachers subject knowledge often inadequate • Physical Education Task (PET) inappropriately assessed.

19. Moderation of Internal Assessment • Phase 1: Findings • Internal Moderation • Moderation instruments – compliance vs. quality • School moderation – seriously neglected • Monitoring vs. Moderation • Feedback and Support – general support by subject advisors. Very little developmental feedback.

20. Moderation of Internal Assessment • Phase 1: Areas of Good Practice • Use of Common Tasks • Diagnostic Analysis to improve performance • Compliance with Internal Assessment protocols.

21. Moderation of Internal Assessment • Phase 2 (October/November 2012)- focus on learner evidence and teacher files.

22. Moderation of Internal Assessment • Phase 2: Findings • Quality and Standard of Moderation • Brief unclear – focus on marking only and not on quality of task. • Shadow marking vs. moderation • Marking of the Tasks • Marking tools – not applied consistently • Teachers subject knowledge often inadequate • Challenges with marking extended writing and research projects. • Marking too lenient

23. Moderation of Internal Assessment • Phase 2: Findings • Learner performance • Difficulty with higher-order questions • Ability to express/respond in English a determining factor • Extremely poor performance in certain subjects e.g. Maths, LO Common Assessment Task, • Evidence suggests that very little effective teaching and learning is taking place

24. Moderation of Internal Assessment • Phase 2: Findings • Internal Moderation at School, Cluster and District Levels. • Varying quality of Moderation • Limited moderation of candidates work – shadow marking • Little evidence of moderation of tasks. • Lack of mark sheets or incorrect recording of marks.

25. Moderation of Internal Assessment • Phase 2: Areas of Good Practice • Use of Common Tasks • Diagnostic Analysis to improve performance • Compliance with Internal Assessment protocols. • Evidence of moderation at the various levels. • Introduction of National Moderation

26. Monitoring of Examinations • State of readiness • Conduct of examinations • Marking

27. Monitoring of Examinations • State of readiness • Findings • All provinces have examination systems in place • Vacant posts and use of contract staff is a major concern.

28. Monitoring of the writing phase. Findings • General Management of the examinations • Generally examinations conducted in line with policy • Isolated instances of non-compliance (suitability of venue, noise, security, contingency plans) • Shortage of question papers and answer books • Provincial Monitoring

29. Monitoring of the marking phase

30. Monitoring of the marking phase Findings • In general marking processes in line with policy • Concern with lack of security at some marking centres • Concern with the appointment and training of markers • Concern with the quality and standard of marking and internal moderation. • Number of marking centres – security risk

31. PURPOSE • Moderation of marking determines the standard and quality of marking and ensures that marking is conducted in accordance with agreed practices • Umalusi engages the following during the moderation of marking 1. Pre-marking/memorandum discussion: centralised memo discussions recommended - this will ensure consistency across marking centres 2. Moderation of marking (off-site and on-site) Verification of marking

32. Verification of marking • Memoranda discussion meetings • Approved and signed off finalized memoranda/ marking guidelines for all the NSC subjects • Concern with non attendance and pre- memo discussion preparations • Concern with timing and scheduling of memo discussions. • Practice of marking and training in the memo discussions impacts positively on standard of marking.

33. Verification of marking Findings Findings • Adherence to Marking memo • Marking generally accurate – problem with marking extended writing and essays

34. Verification of marking • Quality of marking varied. Question marking improved consistency. • Competency of markers to mark interpretation type questions. • Internal moderation • No unauthorised changes to marking guidelines reported • Training of chief markers and internal moderators at memo discussions having a positive impact on marking.

35. Standardisation and Verification of Resulting • Provision of GENFETQA – Council may adjust raw marks. • International practice – large scale assessment systems • Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance. • Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions

36. Statistical moderation Standardisation • 58 subjects standardised • Raw marks accepted: 41 subjects • Moderated Upward : 5 subjects • Moderated Downward : 12 subjects

37. Verification of the Resulting Process

38. Examination Irregularities • The majority of irregularities were of a technical nature and these were reported to Umalusi according to the established channels. • Some irregularities were as a result of registration-related problems, e.g. candidates nor appearing on mark sheets, some registered for incorrect subjects. • Umalusi represented on NEIC Major irregularities: • Alleged leakage of Question Papers in KZN

39. WHAT CAN WE LEARN FROM THE NSC RESULTS? STANDARDISATION DECISIONS DBE NSC 2012

50. Quality Assurance of the IEB 2012 National Senior Certificate Examination Vijayen Naidoo – Sen. Manager : Quality Assurance of Assessments