The future of Assessments Lessons learned internationally

1. The future of AssessmentsLessons learned internationally Washington, 9 March 2010 Andreas SchleicherHead, Indicators and Analysis DivisionOECD Directorate for Education

2. The future of assessments Or the Alchemists’ Stone? The Holy Grail? • Know why you are looking • You cannot improve what you cannot measure • The yardstick for success is no longer just improvement by national standards but the best performing education systems globally • Know what you are looking for • A new assessment culture • Responsive to changing skill requirements • Capitalising on methodological advances • Not sacrificing validity gains for efficiency gains • Know howyou will recognise it when you find it • Gauging predictive validity • Impact on improving learning and teaching • Implications and lessons learned .

4. A world of change – higher education Expenditure per student at tertiary level (USD) Cost per student Graduate supply Tertiary-type A graduation rate

5. A world of change – higher education Expenditure per student at tertiary level (USD) United States Cost per student Finland Graduate supply Tertiary-type A graduation rate

6. A world of change – higher education Expenditure per student at tertiary level (USD) Australia Finland United Kingdom Tertiary-type A graduation rate

7. A world of change – higher education Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

8. A world of change – higher education Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

9. A world of change – higher education Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

10. A world of change – higher education Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

11. A world of change – higher education Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

12. A world of change – higher education Expenditure per student at tertiary level (USD) United States Australia Finland Tertiary-type A graduation rate

13. Latin America then… Hanushek 2009

14. Latin America then and now… Hanushek 2009

15. Latin America then and now… Why quality is the key Hanushek 2009

16. Know what you are looking for The Holy Grail was a well-described object, and there was only one true grail…

17. Schooling in the medieval age: The school of the church

18. Schooling in the industrial age: Uniform learning

19. Schooling in the industrial age: Uniform learning The challenges today: Universal quality Motivated and self-reliant citizens Risk-taking entrepreneurs, converging and continuously emerging professions tied to globalising contexts and technological advance

20. How the demand for skills has changedEconomy-wide measures of routine and non-routine task input (US) Mean task input as percentiles of the 1960 task distribution The dilemma of assessments: The skills that are easiest to teach and test are also the ones that are easiest to digitise, automate and outsource (Levy and Murnane)

21. Education today needs to prepare students… … to deal with more rapid change than ever before… … for jobs that have not yet been created… … using technologies that have not yet been invented… … to solve problems that we don’t yet know will arise • It’s about new… • Ways of thinking • involving creativity, critical thinking, problem-solving and decision-making • Ways of working • including communication and collaboration • Tools for working • including the capacity to recognise and exploit the potential of new technologies • The capacity to live in a multi-faceted world as active and responsible citizens.

22. Mathematics in PISA The real world The mathematical World Making the problem amenable to mathematical treatment A mathematical model A model of reality Understanding, structuring and simplifying the situation Using relevant mathematical tools to solve the problem A real situation Validating the results Mathematical results Real results Interpreting the mathematical results

23. By whom? • Evaluators • Users of feedback • Evaluation agencies National assessment systems differ… Student assessment Classroom • How? Methods and procedures, mix of criteria and instruments • Mapping of feedback to different units • For what? E.g. • Accountability • Improvement Teacher appraisal Who is assessed School School evaluation System System assessment • What? • Inputs • Processes • Outcomes • With whom? • Agents involved

24. Assessment culturesPutting the pressure on top of the education system is the easy part, building capacity is harder Participative/internal Interactive, reflective, critical friend Survey Summative Formative Standardised assessment Inspectorate Administrative external

25. …but there are global trends • Understanding what the assessment reveals about students’ thinking to shape better opportunities for student learning • Responding to assessments can enhance student learning if tasks are well crafted to incorporate principles of learning • Capitalise on improved data handling tools and technology connectivity to combine formative and summative assessment interpretations for a more complete picture of student learning • Multi-layered, coherent assessment systems from classrooms to schools to regional to national to international levels that… • Support improvement of learning at all levels of the education system • Are largely performance-based • Make students’ thinking visible and allow for divergent thinking • Are adaptable and responsive to new developments • Add value for teaching and learning by providing information that can be acted on by students, teachers, and administrators • Are part of a comprehensive and well-aligned continuum, communicate what is expected and hold relevant stakeholders accountable . • Integrate, synthesize and creatively apply content knowledge in novel situations • Activate students as owners of their own learning and activate students as learning resources for one another

26. Know how you will recognise it when you find it The Alchemists’ stone was to be recognised by transforming ordinary metal into gold…

27. Increased likelihood of postsec. particip. at age 19/21 associated with PISA reading proficiency at age 15 (Canada)after accounting for school engagement, gender, mother tongue, place of residence, parental, education and family income (reference group PISA Level 1) Odds ratioCollege entry School marks at age 15 PISA performance at age 15

28. Relationship between test performance and economic outcomesAnnual improved GDP from raising performance by 25 PISA points Percent addition to GDP

29. Implications and lessons learned The medieval Alchemists’ followed the dictates of a well-established science but that was built on wrong foundations The search for the Holy Grail was overburdened by false clues and cryptic symbols

30. From assessment-inhibited practice towards outcome driven reform Strong focus on processes Integrated quality management Good willand trust Weak outcome-based management Strong outcome-based management External control, uninformed prescription Deprivation Weak focus on processes

31. Some criteria used in the world • Coherence • Built on a well-structured conceptual base—an expected learning progression—as the foundation both for large scale and classroom assessments • Consistency and complementarity across administrative levels of the system and across grades • Comprehensiveness • Using a range of assessment methods to ensure adequate measurement of intended constructs and measures of different grain size to serve different decision-making needs • Provide productive feedback, at appropriate levels of detail, to fuel accountability and improvement decisions at multiple levels • Continuity • A continuous stream of evidence that tracks the progress of both individual students .

32. Understanding learning progressions • Learning targets • Defining what mastery means for a given skill level • Progress variables • Delineate a pathway that characterise the steps that learners typically follow as they become more proficient • Evaluation of students reasoning in terms of the correctness of their solutions as well as in terms of their complexity, validity and precision • Levels of achievement • Describing the breadth and depth of the learner’s understanding of the domain at a particular level of advancement • Learning performances • The operational definitions of what student’s understanding would look like at each of the stages of progress . Wilson, ATC21S

33. OECD Level 2 OECD Level 6 • Identifying • Recognising issues that can be investigated scientifically • Identifying keywords in a scientific investigation • Recognising the key features of a scientific investigation • Explaining • Applying knowledge of science in a situation • Describing or interpreting phenomena scientifically or predicting change • Using evidence • Interpreting scientific evidence and drawing conclusions • Identifying the assumptions, evidence and reasoning behind conclusions Students can determine if scientific measurement can be applied to a given variable in an investigation. Students can appreciate the relationship between a simple model and the phenomenon it is modelling. Students can demonstrate ability to understand and articulate the complex modelling inherent in the design of an investigation. • Context • - Personal • Social/public • Global • Competencies • Identify scientific issues • Explain phenomena scientifically • Use scientific evidence Students can recall an appropriate, tangible, scientific fact applicable in a simple and straightforward context and can use it to explain or predict an outcome. Students can draw on a range of abstract scientific knowledge and concepts and the relationships between these in developing explanations of processes • Knowledge • Knowledge of science • Knowledge about science • Attitudes • -Interest in science • -Support for scientific enquiry • -Responsibility Students demonstrate ability to compare and differentiate among competing explanations by examining supporting evidence. They can formulate arguments by synthesising evidence from multiple sources. Students can point to an obvious feature in a simple table in support of a given statement. They are able to recognise if a set of given characteristics apply to the function of everyday artifacts.

34. Some methodological challenges • Can we sufficiently distinguish the role of context from that of the underlying cognitive construct ? • Do new types of items that are enabled by computers and networks change the constructs that are being measured ? • Can we drink from the firehose of increasing data streams that arise from new assessment modes ? • Can we utilise new technologies and new ways of thinking of assessments to gain more information from the classroom without overwhelming the classroom with more assessments ? • What is the right mix of crowd wisdom and traditional validity information ? • How can we create assessments that are activators of students’ own learning ? Wilson, ATC21S

35. High policy value A real-time assessment environment that bridges the gap between formative and summative assessment . Quick wins Must haves Examine individual, institutional and systemic factors associated with performance Extending the range of competencies through which quality is assessed Monitor educational progress Measuring growth in learning Low feasibility High feasibility Establish the relative standing of students and schools Assuming that every new skill domain is orthogonal to all others Money pits Low-hanging fruits Low policy value

36. Getting the sequencing right Phases of development Adequate  Good Poor  Adequate Good  Great Main focus of assessment • Tackling underperformance • Transparency . • Spreading best practice • World class performance. • Continuous learning and innovation . Role of government • Regulating . • Capacity-building • Prescribing . • Justifying • Enabling • Incentivising . Role of professions • Implementing • Accepting evidence • Adopting minimum standards • Accommodating • Evidence-based • Adopting best . practice • Leading • Evidence-driven • Achieving high reliability and innovation . • Principled • Strategic partnership • Negotiated • Pragmatic . • Top-down • Antagonistic . Nature of relationship between government and professions • Steady improvement • Growing public satisfaction . • Consistent quality • Public engagement and co-production . Main outcomes • Improvement in outcomes • Reduction of public anxiety.

37. Thank you ! www.oecd.org; www.pisa.oecd.org All national and international publications The complete micro-level database email: Andreas.Schleicher@OECD.org Twitter: @SchleicherEDU … and remember: Without data, you are just another person with an opinion