Fairness as an issue in school-based assessment

University of Hong Kong Faculty of Education Inaugural Seminar SeriesEnglish Language School-based Assessment: Integrating Theory and Practice, Jan 9th 2006 Fairness as an issue in school-based assessment Liz Hamp-Lyons, Honorary Professor Faculty of Education, University of Hong Kong

Changing assessment cultures • The official adoption in Hong Kong of the UK Assessment Reform Group’s (1998) distinction between assessment for learning, and assessment oflearning has stimulated the beginnings of a major paradigm shift • Hong Kong schools are moving from a culture of testing to a learning and assessment culture (Hamp- Lyons 1999; in press).

Changing assessment cultures in Hong Kong http://cd.emb.gov.hk/basic_guide/BEGuideeng0821/chapter05.html “Based on the beliefs that every student is unique and possesses the ability to learn, and that we should develop their multiple intelligences and potentials … there should be a change in assessment practices and schools should put more emphasis on 'Assessment for Learning' as an integral part of the learning, teaching and assessment cycle …

Changing assessment cultures in Hong Kong … In other words, teachers should use assessments (e.g. as simple as effective verbal questioning, observation of student behaviour) and provide immediate feedback to enhance student learning in everyday classroom lessons. The focus is on why they do not learn well and how to help them to improve rather than just to use assessments to find out what knowledge students have learned.”

Theory and practice • The HKU Faculty of Education’s SBA team is working on “an evolving but coherent programme of research and evaluation, professional development, and system-wide support” (Davison 2005) • Overall, the team’s work in SBA aims to carry out both basic research (addressing fundamental theoretical problems in language assessment) and applied research (responding to the needs of education communities)

The HKCEE English SBA initiative • SBA is integrated into the teaching and learning process, with teachers involved at all stages of the assessment cycle:  planning the assessment programme identifying and developing appropriate assessment tasks  overseeing the assessment process  making final judgments

Fairness: theory and practice • In theory, fairness in an assessment for learning context means that every learner has the highest quality opportunity possible for demonstrating that/what s/he has learned • However, theory of what fairness is remains underdeveloped and fragmented; different kinds of theorists take different positions

Contexts for “fairness” • In practice, fairness in an SBA context means that the right and responsibility of being fair lies with the teacher (see slide 4) • However, in practice different stakeholders have different understandings and different expectations of what makes something “fair

Fairness—the Hong Kong context • Research studies have found that assessment innovation was severely constrained by traditional school culture and by teacher, parental and student expectations (Cheung & Ng 2000; Carless 2001; Adamson & Davison 2003; Davison & Tang, 2003; Hamp-Lyons, Chen & Mok 2000). “Many teachers reported they cannot ‘assess’, only ‘mark’. They feel unable to make a difference in teaching and learning, to respond to individual needs, because of community expectations of convergence and commonality. Teachers feel their assessment processes are expected to change, without fundamental purposes being explicitly challenged. Such role conflict results in increasing stress and a decline in perceived teacher expertise.”

Teachers’ understandings of fairness R: How? How is it fair? T1: It can help the students better. The teachers are here to help. It is more encouraging because it lasts for a period.

Teachers’ understandings of fairness T1: All the ideas are good but still there are loop-holes. The only thing I’m afraid is what if I am the assessor and my relatives are going to be the students, so how can we check or ensure that my relatives will not be assessed by me and there will be absolute fairness. I think it’s a concern to the public.

Teachers’ understandings of fairness • T2: I found HK students really concerned about the issue of fairness as well, so I mean…. evaluation is going to have to be as standardized as possible, I think it is going to be a big problem for a lot of teachers and students to try, you know, not just among different schools but even within one school, different classes… how do you standardize it in a way that’s fair to everyone?

Teachers’ understandings of fairness • R: What are you trying to standardize? • T2: Wow, just the assessment criteria, because as you said it does count. And I found the HK students as soon as they know these are real marks and these marks are really counted for something, then they get motivated right away… first of all we have to go through some kind of learning curricula adjustment to get the teachers and the students used to this sort of style and I mean that will be expected.

Teachers’ understandings of fairness • T2: …such a large window where is not so fair, right, if you know what I’m trying to say, the differences between the schools and different banding of schools and then even schools like ours, the difference between the classes.

Fairness—the system? • T3: Yeah, I think, you know, I think basically the EMB whenever they need to promote certain changes to the education system, they should really think of, you know, is there really a need to change? If there is a need to change, think of how to make the changes smoother from one system to another and also what they can do, what they can do for both the teachers and the students because you know, the teachers are working on the front line, right?

T4: As I have mentioned earlier, I would say as a teacher, a school teacher, examination in the school usually shares two purposes. One is assessing them; the other is teaching... In fact every time I teach them, not just assessing them. Because after their examination, I give them some feedback, right after the examination, group after group. It really helps them to improve and so that’s more… still I think the SBA component is better because it can encourage students to do better. In a way, I think it will affect the atmosphere of the school, the learning atmosphere. The students think that if they learn harder, I mean they can get some marks in advance. And I think it is the right direction.

The need for managed change • Teachers cannot be expected to change their culture of assessment without the support of their panel, the school Principal, the EMB/CDI and the HKEAA • Teachers also need the support of targeted professional development in new assessment cultures • Messages about the purpose and value of change also need to reach parents and the public

Constraints on teachers’ concepts of fairness • These constraints stem partly from the inadequate attention that is paid to educational assessment in professional training (not only in Hong Kong but almost world-wide), but also • From outside influences: as Pryor and Lubisi (2002) report in research on AfL in Ghana and South Africa, “worry is extrinsic rather than intrinsic.”

What is fairness? • Stobart (2005) makes the point that fairness is fundamentally a sociocultural, rather than a technical, issue. He says that • Fair assessment cannot be considered in isolation from both the curriculum and the educational opportunities of the students. • Fairness is “a justice that goes beyond acting in agreed upon ways and seeks to look at justice of the arrangements leading up to and resulting from those actions.” (p.1)

What the 2007 English SBA does… • With school-based assessment and assessment for learning, the teacher’s role is foregrounded. • The teacher is mediator in the assessment process. • An assessment that puts a teacher and a known student together, that encourages mediation, also encourages diagnostic, interactive and collaborative characteristics in the assessments, even those for summative purposes. • The guidelines booklet Introduction to the School-based Assessment Component booklet (SBA Consultancy Team, 2005, commissioned by the HKEAA) marks a shift in the assessor's role from that of an external "tester" with no responsibility to the student other than to make a reliable judgment, to that of a mediator and facilitator whose responsibility is to see that every student has the best possible opportunity.

In assessment for learning, the teacher becomes a mediator and facilitator who is: • sensitive to different learners’ levels and needs • able to make effective use of questioning, prompts and probes to evaluate as well as promote student learning • able to help lower proficiency students to demonstrate their full range of oral skills • able to challenge higher proficiency students. • This form of assessment is commonly known as dynamic assessment (Anton 2003; Lantolf and Poehner 2004) or educative assessment (Wiggins 2005), and is grounded in Vygotsky’s theory of the Zone of Proximal Development (ZPD) (1978).

Areas for clarification-1 • In a questionnaire to SBA teacher trainers, the question on which they were least agreed among themselves was: “To be fair to students, all students should be given the same amount of time for their task.” (SD 1.61 on 6-point scale)

Views of fairness

Why is equal not fair? • Within assessment for learning, fair assessment is defined as assessment that provides the best opportunities for all, that provides checks and balances to help all participants in the system feel that they and their peers are acting fairly or being assessed fairly

Why is equal not fair? • Gender or age factors • Confidence vs shyness • Self-centred vs other-centred • ‘equality’ is a myth, a goal that is never attained: research shows that all oral language assessors vary according to candidate characteristics (e.g. Brown 2003; Iwashita 2004)

Areas for clarification-2 • The other questions where SBA teacher trainers’ views were very varied was: • “The main goal of the standardization session is to check the accuracy of the teachers’ marking.” • (SD 1.58 on 6-point scale where 1=Disagree)

Standardization is not about ‘accuracy’---a • ‘standardisation’ is one of the areas of checks and balances provided in the HKEAA English SBA system • Teachers meet together, look at/listen to/discuss student oral samples, the tasks students have done, etc • They share their scores and talk about why they think a sample is at a level on each domain

Standardization is not about ‘accuracy’---b • Sharing and talking are key processes that lead to understanding and to common ground • Therefore a teacher may choose to adjust a score after such a discussion • But the focus of the meetings is collegial support and sharing, not checking each others’ ‘accuracy’

Changing practice, changing theory • Following Black et al (2003) the HKEAA’s English SBA innovation is changing theory by changing practice • Notions of fairness are being expanded and redefined • The overpowering influence of the narrow concept of reliability is being replaced by a more appropriate one -- trustworthiness • but the system has much to do to explain this to all levels of the community

In conclusion • Messick (1999) discusses issues of: • Access • Assessments that restrict the access to only some of a student population are not fair • Quality • Assessments that are not of validated high quality are not fair • Student development • Assessments that do not provide means for students to develop in the knowledge/skill assessed are not fair • Public policy • Assessments that exist within an inadequate public policy environment are unlikely to be fair

The fundamental fairness • “Overall the purpose of assessment is to improve standards, not merely to measure them.” (Review of secondary education in England, 1993-1997. Government Office for Standards in Education [OFSTED] 1998).

Fairness as an issue in school-based assessment