Application of a DQAF for Education Statistics

1. Application of a DQAF for Education Statistics Brian Buffett (b.buffett@uis.unesco.org) Conference on Data Quality for International Organisations Newport, Wales April 27 - 28, 2006

2. Outline • Assumptions • Introduction • Overview of the DQAF • The DQAF in practice • Observations & Summary

3. Assumptions • Familiarity with the concept of Data Quality Assessment Frameworks; • Awareness of the IMF DQAF;

4. Introduction 1999 ………. UNESCO Institute for Statistics (UIS) was established 2001 ………. UIS moved from Paris to Montreal, Canada 2003 ………. Statistical Capacity Building Programme (SCB) introduced UIS Mission: • Internationally comparable statistics in UNESCO’s areas of competence: Education; Science & Technology; Culture; Communication Currently: • 100 staff members in Montreal • Regional Advisors in: Chile, Senegal, Thailand, Ethiopia, Samoa

5. Why a Statistical Capacity Building Programme? • Statistical Capacity Building is one of the four Main Lines of Action in UNESCO Institute for Statistics statutes. • The SCB programme serves two purposes: • It supports member states to meet their own needs for production and use of statistics in UNESCO domains. • It supports primary UIS data programmes. • Since countries are the source of UIS primary data – improving country data is an essential ingredient for improved international data.

6. Why an Education DQAF? • The SCB programme needed a mechanism in order to efficiently and effectively engage countries and assess the statistical systems within education ministries. • Within education ministries, what was often lacking was not awareness of the need for better data and statistics – but an awareness of what the problems were and a road map for how to go about addressing them. • The UIS desired a broad framework that focussed on the quality-related features of the governance of statistical systems, their core statistical processes, and their statistical products.

7. The IMF DQAF • The IMF DQAF is not limited solely to timeliness and accuracy • Six dimensions to the IMF DQAF: • Prerequisites of quality • Integrity • Methodological soundness • Accuracy and reliability • Serviceability • Accessibility

8. Why extend the IMF DQAF? • The DQAF seemed to meet the overall requirements; • The work was reduced to a fraction – and domain specific; • Cost and timelines to implement were attractive; • The IMF was willing;

9. Extending the DQAF to Education • Developed in 2003 in collaboration with the World Bank. • Addresses: • International Standards and Classifications (ISCED) • Best practices and guidelines specific to education • Verifies statistical system measures and reports on: • Structure and normative characteristics of education system • Supply of education • Demand for education • Quality of learning outcomes • School environment • …

10. The DQAF in Practice • Used to diagnose the situation of national information systems on education, paying particular attention to national information needs. • These diagnoses are a major element to devise action plans to strengthen national capabilities on education statistics. • International reporting requirements are addressed but not the primary objective. • Flexibility in developing action plans: • If there are significant problems related to international data reporting, ISCED is a critical element; • if the major problems are related to nationally-specific challenges, other items are addressed.

11. The DQAF in Practice • How have the diagnostics been carried out? • Weighted the DQAF components and developed a scoring guide; • Development of common methods and best practices; • Scoring is to a significant extent an expert judgement. Diagnostic missions carried out by a small number of trained staff using common methods. • Regional activities can be facilitated by ensuring coherence across countries

12. The DQAF in Practice • In Latin America and the Caribbean region, by the end of 2006: • Completed in Honduras, Ecuador, and El Salvador; are in revision in Costa Rica, and Nicaragua, and being prepared for Guatemala, Uruguay, Paraguay, Peru and Colombia; • Are being prepared for a similar number of Caribbean countries. • Will result in systematic diagnoses under a common framework for half of the region. • The identification of common challenges will: • permit grouping of countries; • provide the basis for country-to-country cooperation; • Will have the necessary information to support national as well as regional purposes.

13. Summary – the Country Perspective • A useful tool to help strengthen the country’s statistical system by identifying the strengths and weaknesses of the system as well as areas to be improved. • Some results: • adoption of new questionnaires better responding to country information needs; • improved collection methodology – including training of respondents; • more timely completion and return of completed questionnaires; • more efficient data capture – with edits to ensure data quality; • more efficient processing of the data and production of outputs; • improved access to data; • training of statisticians and policy makers in use and interpretation of the data.

14. Summary – the UIS Perspective • Reduced resource costs, timeframe, and skills required for framework development; Able to focus resources and efforts on subject-matter specificities; • The framework and diagnostic method are effective; • Initial results have been achieved more rapidly due to this approach; • Country quality reports on education can be comparable - providing more flexible approaches to capacity building and increasing country-to-country cooperation; • Common best practices need to be followed, such as: • Country ownership; • Broad involvement; • Assessments combined with audits; • Consistent application of scoring guides. • UIS will benefit from future IMF work on the DQAF;

15. Summary – the International Perspective • An example of collaboration and reuse of existing methods – adapted to International Org. environment/needs; • Factor country needs and situation into any approach; • Statistical activities outside of NSO’s can benefit from the same practices as NSO’s; • A broad definition of quality is important;

16. Thank You