Transforming assessment through the tenets

1. Transforming assessment through the tenets Chris Rust and Liz McDowell 8 May 2013

2. This session aims to offer: • Some thoughts on the nature of assessment as a transformative process to enhance student learning; • An overview of the six tenets; • A review of how tenets three and four integrate with the other tenets; • Some practical suggestions to support your work based on tenets 3 & 4; • Opportunities to develop your work and share ideas and issues so you can start to make some meaningful changes at your institutions.

3. Background to ‘A marked improvement’ • Takes an evidence-informed approach; • Builds on expertise, perspectives, and previous work; • Aims to take a radical approach, recognising that it is time for significant reappraisal of assessment policy and practice (ASKe, Weston Manor Manifesto); • Encourages that assessment be seen as an integral part of the learning experience.

4. Why change is needed • QAA subject reviews • Burgess Report - “system no longer fit for purpose” (p5) • QAA - “…it cannot be assumed students graduating …. will have achieved similar standards” (2007) • Media accusations of dumbing down and grade inflation • National Student Satisfaction Survey • “the Achilles’ heel of quality” (Knight 2002a, p. 107) • Summative assessment practices “in disarray” • (Knight 2002b, p. 275) • “Broken” (Race 2003, p. 5) • “There is considerable scope for professional development in the area of assessment” (Yorke et al, 2000, p7)

5. Why change is needed (T1) • “The types of assessment we currently use do not promote conceptual understanding and do not encourage a deep approach to learning………Our means of assessing them seems to do little to encourage them to adopt anything other than a strategic or mechanical approach to their studies.” • (Newstead 2002, p3) • Students become more interested in the mark and less interested in the subject over the course of their studies (Newstead 2002, p2). • Many research findings indicate a declining use of deep and contextual approaches to study as students’ progress through their degree programmes • (Watkins & Hattie, 1985; Kember et al, 1997; Richardson, 2000; Zhang & Watkins, 2001)

6. Tenet 1 “The debate on standards needs to focus on how high standards of learning can be achieved through assessment. This requires a greater emphasis on assessment for learning rather than assessment oflearning”

7. Assessment for learning Adapted from Sambell, McDowell and Montgomery (2012, p5)

8. T1 Key questions: To what extent and how do you evidence this? • Are you confident that assessment tasks demand high standards of learning? • Is assessment for learning given emphasis in relation to assessment of learning? • Do you ensure an appropriate balance between formative and summative assessment? • Is assessment and feedback planned within and across a programme to ensure appropriate student preparation and practice before summative assessment takes place? (From ‘A marked improvement’)

9. Why change is needed (T2) • Our current systems focused on marks and grades are not working: • Belief that it is possible to distinguish the quality of work to a precision of one percentage point (Elander & Hardman, 2002); • If the scale is not quantitative (i.e. 60 = 1.5 x 40), standard arithmetic operations on the grades are illegitimate (Dalziel, 1998) • Use of ‘norm’ referencing (formal or informal) • “The normal curve is a distribution most appropriate to chance and random activity.  Education is a purposeful activity and we seek to have students learn what we would teach.  Therefore, if we are effective, the distribution of grades will be anything but a normal curve.  In fact, a normal curve is evidence of our failure to teach.” (Bloom et al, 1971) • The combination of scores, which obscures the different types of learning outcome represented by the separate scores; • Transactional and bestowed credits & debits (Sadler, 2009)

10. Why change is needed (T2) • Our current systems focused on marks and grades are not working (contd): • Belief that consistency can be achieved through conformity, and simple numerical rules (e.g. level 1 essay 3,000 words, level 3 essay 5,000; or no more than two pieces of assessment per module); • The combination of scores, which obscures the different types of learning outcome represented by the separate scores; • The distortion of marks by: • the type of assessment (e.g. c’work c.f. examination); • the actual subject discipline/s studied (Yorke, 1997; Yorke et al, 2000, Bridges et al, 2002) • The distortion of resulting degree classifications by the application of idiosyncratic institutional rules • (e.g. Armstrong et al, 1998).

11. Tenet 2 “When it comes to the assessment of learning, we need to move beyond systems focused on marks and grades towards the valid assessment of the achievement of intended programmeoutcomes”

12. T2 Key questions: To what extent and how do you evidence this? • Is there an emphasis on assessment for learning over systems focused on marks, grades and reliability? • Does the assessment design process ensure valid assessment of the intended learning outcomes? • Is there a trade off between reliability and validity of assessment? • Are assessment decisions in relation to design, development and variety made within a programme context and focused on programme learning outcomes? • (From ‘A marked improvement’)

13. Why change is needed (T3) • Regulative and logical criteria “standards can be defined in terms of well-defined outcomes” (Sadler, 1987, p. 70) • Prescriptive and constitutive criteria refer to matters of degree and “It would be difficult or impossible to guess the educational level at which they are applicable… ” (Ibid) • Verbal level descriptors are inevitably ‘fuzzy’ (Ibid) • Such types of criteria are often interdependent and can only be assessed using holistic/professional judgement (Sadler, 2008) • Meaningful understanding of standards requires both tacit and explicit knowledge (O’Donovan et al. 2004) • “We can know more than we can tell” (Polanyi, 1998, p.136)

14. Tenet 3 “ Limits to the extent that standards can be articulated explicitly must be recognised since ever more detailed specificity and striving for reliability, all too frequently, diminish the learning experience and threaten its validity. There are important benefits of higher education which are not amenable either to the precise specification of standards or to objective assessment”

15. Two simple implications of Tenet 3 Accept that criteria can’t capture everything (the tacit) so do not create marking systems with grades/marks allocated to individual criteria to be added up to form final mark Accept that not everything can or needs to be summatively marked with a mark or a grade but may nevertheless be a very worthy activity and benefit from formative qualitative feedback, however subjective

16. Why change is needed (T4) • ‘making sense of the world’ is a social and collaborative activity (Vygotsky, 1978) • Criteria are socially constructed requiring the sharing of tacit knowledge over time (O’Donovan et al, 2004; Rust et al, 2005) • Tacit knowledge is experience-based and can only be revealed through the sharing of experience – socialisation processes involving observation, imitation and practice (Nonaka, 1991) • An indispensable condition for improvement in student learning is that “the student comes to hold a concept of quality roughly similar to that held by the teacher” (Sadler, 1989)

17. Why change is needed (T4) • Passive receipt of feedback has little effect on future performance (Fritz, et al., 2000) • Dialogue and participatory relationships are key elements of engaging students with assessment feedback (ESwAF FDTL, 2007) • It is not enough to make it a better monologue; feedback must be seen as a dialogue (Nicol, 2009) • “participation, as a way of learning, enables the student to both absorb, and be absorbed in the culture of practice” • (Elwood & Klenowski, 2002, p. 246) • The most significant factor in student academic success is student involvement fostered by student/staff interactions and student/student interactions (Astin, 1997) • The only common factor in a study of departments deemed excellent in both research and learning and teaching is high levels of student involvement (Gibbs, 2007)

18. Tenet 4 “Assessment standards are socially constructed so there must be a greater emphasis on assessment and feedback processes that actively engage both staff and students in dialogue about standards. It is when learners share an understanding of academic and professional standards in an atmosphere of mutual trust that learning works best”

19. Assessment design & development of explicit criteria Active engagement with feedback Active engagement with criteria Completion and submission of work Rust C.,O’Donovan B & Price., M (2005) Tutor discussion of criteria Staff Assessment guidance to staff Marking and moderation Explicit Criteria Students

20. Engaging students with criteria • Get students actively using the criteria through a developmental combination of: • Marking exercises • Self-assessment • Peer-feedback • Peer-assessment • Possibly creating and negotiating criteria

21. Saving time – a range of approaches to feedback need to be in place • Giving generic feedback within 24 hours on the VLE • Offering face- to-face feedback with whole group/small groups • Using peer feedback in groups • Using technologies • see http://tinyurl.com/tfaproject: e.g MCQ quizzes • Aural feedback, e.g. Camtasia software • see http://sites.google.com/site/soundsgooduk/ • Encouraging students to use self feedback • Helping students to recognise that feedback can come from many different sources (including themselves) and given in different ways

22. At the very simplest, need two conceptual shifts Self and peer assessment need to be seen as essential graduate attributes (i.e. learning outcomes themselves, rather than processes) Feedback needs to be seen as a dialogue (rather than a monologue) … with an explicit intention to bring students into the community of assessment practice

23. Assessment design & development of explicit criteria Active engagement with feedback Active engagement with criteria Completion and submission of work Rust C.,O’Donovan B & Price., M (2005) Tutor discussion of criteria Staff Assessment guidance to staff Marking and moderation Explicit Criteria Students

24. Why change is needed (T5) • Important aspects of complex, high-level learning outcomes can only be achieved when students are allowed time to ‘come to know’ the standards in use by the community: • Slowly learnt academic literacies require rehearsal and practice throughout a programme (Knight and Yorke 2004) • The achievement of high-level learning requires integrated and coherent progression based on programme outcomes • Where there is a greater sense of the holistic programme students are likely to achieve higher standards than on more fragmented programmes (Havnes, p. 2007) • Students need to engage as interactive partners in a learning community, relinquishing the passive role of ‘the instructed’ within processes controlled by academic experts (Gibbs et al, 2004)

25. Tenet 5 “Active engagement with assessment standards needs to be an integral and seamless part of course design and the learning process in order to allow students to develop their own, internalised, conceptions of standards and monitor and supervise their own learning” (for further discussion at third targeted forum, but here as part of a holistic discussion of the tenets as a whole)

26. Why change is needed (T6) • Changes in higher education (e.g. massification, reduced unit of resource, expectations of increased productivity in staff) threaten the ‘health’ of disciplinary communities and their ability to share and exemplify professional judgement. • In the professionalisation of university teachers there has been limited attention paid to professional assessment practice. • Reliance on the external examiner system to mediate standards within the system is misplaced (Newstead and Dennis,1994). • Cf “a college of peers” (Ramsden, 2008) If some aspects of high-level learning can only be assessed using professional judgement then we need to ensure that judgement is indeed – professional!!

27. Tenet 6 “Assessment is largely dependent upon professional judgement and confidence in such judgement requires the establishment of appropriate forums for the development and sharing of standards within and between disciplinary and professional communities” (for further discussion at third targeted forum, but here as part of a holistic discussion of the tenets as a whole)

28. Why change is needed (summary) • The current approach to assessment standards is inadequate; • The quest for reliability is getting in the way of learning; • We should be enabling students to achieve high level, complex learning; • Active involvement in assessment, by staff and students in the learning community, is essential to reach common understandings of assessment standards; • To achieve change there may need to be a review and evaluation of the allocation of time and resources

29. Connecting the tenets Programmes 4 5 Outcomes Local Subject Learning Community Standards Summativeassessment Criteria 2 Formative Beliefs Operations Operations (students and staff) Operations Assessment for learning 1 3 Assessment literacy, professional judgement and disciplinary and professional communities 6

30. Assessment for learning Adapted from Sambell, McDowell and Montgomery (2012, p5)

31. Assessment for learning Which one of these is most important to you? And why? Adapted from Sambell, McDowell and Montgomery (2012, p5)

32. How do we engage students in assessment communities? Understanding standards : exercises, exemplars, feedback, self-assessment ... Being a participant : choice in assessment, student voices, sense of belonging and purpose .....

33. 11.30 – 12.45 Activities 2 • If we want to make changes on the basis of Tenets 3 and 4 – or others: • Who do we need to involve? • Why? What do we want them to do? • How can we involve them?