ww.oecd/edu/evaluationpolicy

1. Evaluation, Accountability and Improvement in Education: Perspectives from the OECD Review on Evaluation and Assessment Frameworks ww.oecd.org/edu/evaluationpolicy Paulo Santiago Senior Analyst, Directorate for Education, OECD Improving Education through Accountability and Evaluation: Lessons from around the World International Conference Rome, 3 October 2012

2. Outline • OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes • Key themes • The rise of educational measurement and indicators development • The growing prominence of accountability as a purpose of evaluation • Establishing links to classroom practices • Building capacity for evaluation and assessment • Fostering synergies within the evaluation and assessment framework

3. 1. OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes

4. OBJECTIVES Purpose: To explore how systems of evaluation and assessment can be used to improve the quality, equity and efficiency of school education. A Review of national approaches to evaluation and assessment in school education [student assessment, teacher appraisal, school evaluation, school leader appraisal, education system evaluation] The Review: • Synthesises research-based evidence on the impact of evaluation and assessment strategies and disseminate this knowledge among countries. • Identifies innovative and successful policy initiatives and practices. • Facilitates exchanges of lessons and dialogue. • Identifiespolicy options for governments to consider. Comprehensive approach: Investigation of each component individually, as well as the coherence of the framework as a whole (including the links between the different components).

5. METHODOLOGY • Analytical strand • Identifying the key questions for analysis and the background information needed from countries • Reviewing the literature and evidence on the impact of evaluation and assessment procedures • Gathering data on countries’ policies and practices (Country Background Reports) • Country Review strand • Country Reviews provide specific advice to individual countries. • OECD-led Review Team with external experts • The scope and focus is determined by the country in consultation with the Secretariat • Synthesis report • Comparative report to analyse policy options and highlight good practices across countries. • Twenty six systems participating • Australia, Austria, Belgium (Flemish Community), Belgium (French Community), Canada, Chile, Czech Republic, Denmark, Finland, France, Hungary, Iceland, Ireland, Italy, Korea, Luxembourg, Mexico, Netherlands, New Zealand, Norway, Poland, Portugal, Slovak Republic, Slovenia, Sweden, United Kingdom (Northern Ireland).

6. 2. Key themes

7. The rise of educational measurement and indicators development Trends: • The growing emphasis on measuring student outcomes; the proliferation of education indicators; and the establishment of education targets Potential: • Student outcomes as the focal point for analysis • Monitoring key student learning outcomes • Formative use of standardised tests • Hold stakeholders accountable Challenges: • Ensuring breadth of performance measures • Securing fair and meaningful comparisons • Avoiding detrimental effects of uses for accountability • Preventing dominance of quantitative over the qualitative Options: • Ensure policy making is informed by high quality measures, but not driven by their availability [the need to complement with qualitative analysis] • Ensure a broad approach to national monitoring • Clarity of purposes for the uses of standardised tests results

8. The growing prominence of accountability as a purpose of evaluation Trends: • Public reporting of school results (greater transparency, reporting requirements); rewards and sanctions on the basis of evaluation and assessment results (e.g. teachers, schools). Potential: • Creation of incentives for improved performance – opportunity to recognise and reward • Identification of underperformance • Facilitates school choice Challenges: • Range of potential detrimental effects (e.g. ‘teaching-to-the-test”, “narrowing of curriculum”) • Accountability function of evaluation not to hinder the improvement function • Conveys a ‘control’, ‘compliance’, ‘measurement’ concept of evaluation • Focus on accountability as a result of a strong top-down national vision of evaluation might constrain the ownership of E&A by school agents Options: • Communicate rationale for evaluation and assessment: objective is improvement • Build on a range of evaluation and assessment components achieving a variety of functions • Publication of quantitative data to be perceived as fair by schools set in a wider set of evidence (of a qualitative nature) • Establish safeguards against overemphasis on student standardised tests (more relevant for whole-school evaluation)

9. Establishing links to classroom practices Trends: • There is often focus on structures, procedures, programmes and resources in a top-down approach Potential: • National agendas for education to be strengthened by consistency of evaluation procedures • Ensuring equity of objectives Challenges: • Evaluation and assessment have no value if they do not lead to the improvement of classroom practice and student learning • Improvement function accomplished more at a local level – need to secure adequate links to the classroom • Risks that evaluation procedures do not place adequate focus on what is arguably the most important area: teaching and learning Options: • Critical to ensure that the evaluation of teaching and learning quality is central to evaluation procedures • Build on teacher professionalism: Teachers to have ownership of student assessment, build teachers’ ability to assess against educational standards, teachers to be involved in school evaluation, emphasis on teacher evaluation for the continuous improvement of teaching practices.

10. Building capacity for evaluation and assessment Trends: Evaluation and assessment policies often introduced with no due attention to capacity development (e.g. competencies for evaluation and assessment) Challenges: Legitimacy of evaluators and accountability procedures The effectiveness of evaluation relies to a great extent on ensuring that both those who design and undertake evaluation activities as well as those who use their results have the proper skills and competencies. Examples of areas for capacity development: standardised test development; formative assessment; assessment against standards; running systems of externally-based student examinations; analytical capacity in education agencies to use system level information; data handling skills of school agents; formal evaluators of individual school agents; competencies for classroom observation. Options: • Sustaining efforts to improve the capacity for evaluation and assessment • Strengthen school leaders’ capacity for school development and instructional leadership • Ensuring support from the centre and identification of best practice • Need for a strong capability at the national level to steer evaluation

11. Fostering synergies within the evaluation and assessment framework Trends: • Most countries do not have an integrated evaluation and assessment framework but instead a series of components operating at different levels that developed independently of each other over time. Potential: • Build synergies • Generate complementarities • Avoid duplication • Prevent inconsistency of objectives Challenges and options: Establishing articulations within the evaluation and assessment framework • Within specific components of the overall E&A framework e.g. teacher appraisal and teacher professional development; self- and external school evaluation. • Between specific components of the overall E&A framework e.g. teacher appraisal, school evaluation and school development; school evaluation and system evaluation; school evaluation and the appraisal of school principals. • Processes to ensure the consistency of E&A procedures Moderation processes for teacher appraisal and teacher-based assessment Clarity of responsibilities across the framework

12. Thank you for your attention! paulo.santiago@oecd.org www.oecd.org/edu/evaluationpolicy