PISA 2009 Evaluating systems to improve education

1. Programme for International Student Assessment PISA 2009Evaluating systems to improve education The yardstick for success is no longer improvement by national standards alone but the best performing education systems

2. PISA 2009 in brief PISA countries in 2003 2000 2001 2006 2009 1998 Coverage of world economy 83% 87% 81% 85% 77% 86% • Over half a million students… • representing 28 million 15-year-olds in 74* countries/economies … took an internationally agreed 2-hour test… • Goes beyond testing whether students can reproduce what they were taught… … to assess students’ capacity to extrapolate from what they know and creatively apply their knowledge in novel situations … and responded to questions on… • their personal background, their schools and their engagement with learning and school • Parents, principals and system leaders provided data on… • school policies, practices, resources and institutional factors that help explain performance differences . * Data for Costa Rica, Georgia, India, Malaysia, Malta, Mauritius, Venezuela and Vietnam will be published in December 2011 Changes in skill demand

3. PISA 2009 in brief PISA countries in 2003 2000 2001 2006 2009 1998 Coverage of world economy 83% 77% 81% 85% 86% 87% • Key principles • ‘Crowd sourcing’ and collaboration • PISA draws together leading expertise and institutions from participating countries to develop instruments and methodologies… … guided by governments on the basis of shared policy interests • Cross-national relevance and transferability of policy experiences • Emphasis on validity across cultures, languages and systems • Frameworks built on well-structured conceptual understandingof assessment areas and contextual factors • Triangulation across different stakeholder perspectives • Systematic integration of insights from students, parents, school principals and system-leaders • Advanced methods with different grain sizes • A range of methods to adequately measure intended constructs with different grain sizes to serve different decision-making needs • Productive feedback, at appropriate levels of detail, to fuel improvement at multiple levels .

4. What 15-year-olds can do

5. High reading performance Average performanceof 15-year-olds in reading – extrapolate and apply … 17 countries perform below this line Low reading performance

6. High reading performance Average performanceof 15-year-olds in science – extrapolate and apply High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low reading performance

7. High reading performance 2009 Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low reading performance

8. High reading performance 2009 Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low reading performance

9. High performing systems often prioritize the quality of teachers over the size of classesContribution of various factors to upper secondary teacher compensation costsper student as a percentage of GDP per capita (2004) Percentage points

10. High reading performance 2009 Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low reading performance

11. High reading performance 2000 Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low reading performance

12. High reading performance 2000 Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Other rapid improvers in reading: Peru, Indonesia, Latvia, Israel and BrazilRapid improvers in mathematics: Mexico, Brazil, Turkey, Greece, Portugal, Italy and Germany Rapid improvers in science: Qatar, Turkey, Portugal, Korea, Brazil, Colombia, Italy, Norway, United States, Poland Low average performance Large socio-economic disparities Low average performance High social equity Low reading performance

13. Changes in performance by type of task Increase percentage correct 2009 2009 2006 2006

14. Changes in percentage of students who reported having a computer at home between PISA 2000 and 2009

15. Changes in computers-per-student ratio between PISA 2000 and 2009

16. Why PISA 2009 included a digital reading assessment • “Digital reading” means reading on screen, mostly via the Internet • Digital reading  reading on line  is an increasingly important form of reading in the 21st century • Digital reading has different features to print reading and makes new demands on readers

17. How proficient are students in digital reading? Tasks at Level 3 require that the reader integrate information, either by navigating across several sites to find well-defined target information, or by generating simple categories when the task is not explicitly stated. Where evaluation is called for, only the information that is most directly accessible or only part of the available information is required. Percentage of students Tasks at Level 5 or above typically require the reader to locate, analyse and critically evaluate information, related to an unfamiliar context, in the presence of ambiguity. They require criteria to evaluate the text. Tasks may require navigation across multiple sites without explicit direction, and detailed interrogation of texts in a variety of formats. Tasks at Level 4 may require the reader to evaluate information from several sources, navigating across several sites comprising texts in a variety of formats, and generating criteria for evaluation in relation to a familiar, personal or practical context. Other tasks at this level demand that the reader interpret complex information according to well-defined criteria in a scientific or technical context. Tasks at Level2 typically require the reader to locate and interpret information that is well-defined, usually relating to familiar contexts. They may require navigation across a limited number of sites and the application of web-based navigation tools such as dropdown menus, where explicit directions are provided or only low-level inference is called for. Tasks may require integrating information presented in different formats, recognising examples that fit clearly defined categories. Students performing below level 2 can scroll and navigate across web pages, as long as explicit directions are provided, and can locate simple pieces of information in a short block of hypertext. Nevertheless, although the digital reading skills of these students are not necessarily negligible, they are performing at levels that are not likely to allow them full access to educational, employment and social opportunities in the 21st century.

18. Differences between boys and girls compared with print reading Gender difference in print reading performance (girls- boys) Girls have a larger advantage in print reading Poland Norway New Zealand Sweden Iceland Austria France Japan Ireland Hungary OECD Average Macao - China Australia Hong Kong -China Korea Belgium Denmark Spain Chile Colombia Girls have a larger advantage in digital reading Gender difference in digital reading performance (girls-boys)

19. Navigation • Navigation is considered to be part of the cognitive process of digital reading • Tracking and analysing the sequences of pages students visit provide insights into effective navigation behaviours in digital reading • The index number of relevant pages visited describes how many of the pages judged to be relevant to a task were accessed while the student worked on that .

20. Number of relevant pages visited and digital reading performance The index number of relevant pages visited describes how many of the pages judged to be relevant to a task were accessed while the student worked on that task

21. ICT use at home for leisure and digital reading performance

22. ICT use at home for schoolwork and digital reading performance

23. ICT use at school and digital reading performance

24. Some conclusions • Identifying effective strategies to teach digital reading skills is an important policy objective • Ability to critically evaluate the quality and credibility of texts, integrate information from multiple texts and – crucially – navigate effectively • ICT use at home for leisure is – up to a point – positively related to performance, navigation skills and self-confidence in completing high-level ICT tasks • Parents and teachers need to encourage students to use computers so that they can improve their navigation skills but also provide guidance on balancing time spent using computers with time for other activities • Digital reading can be a lever to reduce the gender gap • The gender gap in digital reading is much smaller than in print reading, and relates to differences in navigation skills between boys and girls • Reading more and reading with enjoyment promotes better reading, and better reading fosters stronger engagement .

25. Some conclusions • Access to computers has improved significantly, both at home and at school, but important gaps remain between countries and social groups • Strategies that promote wider access to ICT at school can help minimise impact of social background on digital competency gaps, with possible consequences for future employment opportunities • Schools can do more to integrate ICT into learning • Opportunities for students to solve problems using ICT • Reading methods that improve students’ ability to distinguish between relevant and irrelevant material, and to structure, prioritise, distil and summarise texts • ICT can… • Enable students to obtain more regular feedback on their learning processes • Make students more active participants in learning processes in classrooms and tailor these processes to individual students’ needs • Provide students with up-to-date access to the world’s current research and thinking .

26. What does it all mean?

27. A commitment to education and the belief that competencies can be learned and therefore all children can achieve • Universal educational standards and personalisation as the approach to heterogeneity in the student body… • … as opposed to a belief that students have different destinations to be met with different expectations, and selection/stratification as the approach to heterogeneity • Clear articulation who is responsible for ensuring student success and to whom Lessons from PISA on successful education systems

28. Clear ambitious goals that are shared across the system and aligned with high stakes gateways and instructional systems • Well established delivery chain through which curricular goals translate into instructional systems, instructional practices and student learning (intended, implemented and achieved) • High level of metacognitive content of instruction Lessons from PISA on successful education systems

29. Capacity at the point of delivery • Attracting, developing and retaining high quality teachers and school leaders and a work organisation in which they can use their potential • Instructional leadership and human resource management in schools • Keeping teaching an attractive profession • System-wide career development Lessons from PISA on successful education systems

30. Incentives, accountability, knowledge management • Aligned incentive structures • Forstudents • How gateways affect the strength, direction, clarity and nature of the incentives operating on students at each stage of their education • Degree to which students have incentives to take tough courses and study hard • Opportunity costs for staying in school and performing well • For teachers • Make innovations in pedagogy and/or organisation • Improve their own performance and the performance of their colleagues • Pursue professional development opportunities that lead to stronger pedagogical practices • A balance between vertical and lateral accountability • Effective instruments to manage and share knowledge and spread innovation – communication within the system and with stakeholders around it • A capable centre with authority and legitimacy to act Lessons from PISA on successful education systems

31. School autonomy, accountability and student performanceImpact of school autonomy on performance in systems with and without accountability arrangements PISA score in reading

32. Lessons from PISA on successful education systems • Investing resources where they can make most of a difference • Alignment of resources with key challenges (e.g. attracting the most talented teachers to the most challenging classrooms) • Effective spending choices that prioritise high quality teachers over smaller classes

33. A learning system • An outward orientation of the system to keep the system learning, international benchmarks as the ‘eyes’ and ‘ears’ of the system • Recognising challenges and potential future threats to current success, learning from them, designing responses and implementing these Lessons from PISA on successful education systems

34. Coherence of policies and practices • Alignment of policies across all aspects of the system • Coherence of policies over sustained periods of time • Consistency of implementation • Fidelity of implementation (without excessive control) Lessons from PISA on successful education systems

35. The old bureaucratic system The modern enabling system Student inclusion Some students learn at high levels All students need to learn at high levels Curriculum, instruction and assessment Routine cognitive skills, rote learning Learning to learn, complex ways of thinking, ways of working Education reform trajectories Teacher quality Few years more than secondary High-level professional knowledge workers Work organisation ‘Tayloristic’, hierarchical Flat, collegial Accountability Primarily to authorities Primarily to peers and stakeholders

36. High policy value Understanding the instructional context of learning outcomes – linking how students learn with what teachers do Quick wins Must haves Examining individual, institutional and systemic factors associated with quality, equity and efficiency in education Understanding drivers of successful reform trajectories Monitoring educational progress ‘Democratising PISA’ Extending the range of competencies through which quality is assessed The development of PISA PISA 2000 More difficult Less difficult Measuring student learning outcomes in key subjects Electronic delivery of assessments Proliferation of assessment areas . Money pits Low-hanging fruits Moderate policy value

37. High policy value Understanding the instructional context of learning outcomes – linking how students learn with what teachers do Quick wins Must haves Examining individual, institutional and systemic factors associated with quality, equity and efficiency in education Understanding drivers of successful reform trajectories Monitoring educational progress ‘Democratising PISA’ Extending the range of competencies through which quality is assessed PISA 2003 More difficult Less difficult Measuring student learning outcomes in key subjects and establishing the comparative strengths and weaknesses of education systems Electronic delivery of assessments Proliferation of assessment areas . Money pits Low-hanging fruits Moderate policy value