First results from the in-depth surveys on quality assurance frameworks and quality reporting

1. First results from the in-depth surveys on quality assurance frameworks and quality reporting Conference on Data Quality for International Organisations 27-28 April 2006, Newport (UK) Håkan Linden, Statistical Governance, Quality and Evaluation European Commission, Eurostat

2. Background • Key areas of work in the Eurostat project on “the use and convergence of international quality assurance frameworks” sponsored by the CCSA. • Problem statement – quality assurance frameworks: • - lack of use of quality assurance frameworks in most international organisations • - existence of different quality frameworks and benchmarking activities for national statistics reported • Problem statement – quality reporting: • - existence of different tools and practices for collecting quality metadata from the data providers • - existence of different formats to inform the users on the quality of statistics

3. Overview in-depth surveys • Questionnaire templates developed by the task-teams for structuring the work and collecting the state-of-art on quality frameworks and quality reporting activities. • All members of the CCSA included in the surveys • Data collection: 20 February 2006 – 31 March 2006 • 12 international organisations replied – thank you! • First results for this conference • Report with detailed results and recommendations to be prepared for the 2nd CCSA meeting 2006

4. Quality frameworks - the template • Section 6. Quality Assurance Frameworks in place • - brief description of the quality assurance framework • - legal basis • - last update • - coverage of institutional environment, statistical processes and outputs • - quality definition • - quality requirements/ targets • - procedures for evaluation of the adherence to the quality framework • - main strengths and weaknesses of the quality framework • Section 7. If a quality framework not yet in place • - brief description of the current situation of the quality work • - how weaknesses are identified, impact assessed and improvement actions • - main obstacles for development and implementation of a quality framework • Section 8. For all international organisations • - the future plans for quality improvements • - additional comments/ recommendations etc. for this project

5. Quality frameworks - results • 5 (out of 12) organisations have quality assurance frameworks in place • 1 organisation is currently developing an encompassing quality assurance framework • 6 organisations have not yet begun to develop formalised quality assurance frameworks

6. Quality frameworks - in place • “Legal” basis: • - endorsed by (internal) Statistical Policy Group and Statistical Committee • - Data Quality Standard under development • - Article • - Policy document • - Adherence to the “Principles Governing Statistical Activities” • - Recommendation • Last update: 2003 (new update in 2006); May 2004; July 2003; May 2005; 2003.

7. Quality frameworks - in place • Coverage of institutional environment/ settings: • - all statistical activities/ major activities reviewed every 5- years • - statistical data under “direct coordination” but extension of the coverage planned • - 2 organisations actually cover the institutional settings!

8. Quality frameworks - in place • Coverage of statistical processes: • - definition of data requirements • - evaluation of other data currently available • - statistical design and planning • - data and metadata collection • - data and metadata processing • - compilation and estimation • - disclosure control • - data and metadata dissemination

9. Quality frameworks - in place • Coverage of statistical output: • - all • - limited to a few domains • - economic monetary statistics

10. Quality frameworks - in place • Quality requirements/ targets: • (i) Institutional environment/ settings: • - benchmarks for observation of international good practice or standard with reference to for example the UN Fundamental Principles of Official Statistics, the UN Handbook of Statistical Organization, and the Quality Declaration for the ESS • (ii) Statistical processes: • - a set of best practices is defined for each process. Each activity must be compared with these practices • - benchmarks … • - standardisation and harmonisation of tools, reports and development of policies (like revision policy) • (iii) Statistical outputs: • - a set of best practices is defined for each process. Each activity must be compared with these practices • - punctuality (annually agreed timetables) • - regulations

11. Quality frameworks - in place • Procedures in place for the evaluation of the adherence to the framework: • (i) Institutional environment/ settings: • surveillance missions may help to identify institutional and legal issues underlying problems in data quality • audit exercises • self-assessments • peer reviews • annual report on the implementation and protection of confidential information • (ii) Statistical processes: • - quality reviews of ongoing activities on a 5 year rolling basis (self-evaluation of strengths and weaknesses by comparing with best-practices) • (iii) Statistical outputs: • - compliance reports • - regular reports on gaps in statistics (as input for medium-term work-pgm) • - quality reports/ quality profiles

12. Quality frameworks - in place • Strengths: • - Quality reviews (self-evaluation of strengths and weaknesses by responsible persons, comparing their activities with best-practices) on a rolling basis give a good framework for discussing quality problems between domain managers and statistics and IT- experts • - Data quality monitoring at each stage of a statistical production process makes it possible to identify and address data quality problems. Incorporated “feed-back loops” serve as a mechanism for improving data quality • - The implementation of quality framework (phased/ staggered approach) ease the development of new work processes related to data collection and processing, IT, and data dissemination • - Existence of transparent and comprehensive legislative framework for the production of statistics ease the monitoring of data quality • - Quality manager appointed to coordinate all quality work

13. Quality frameworks – in place • Weaknesses: • - The resources allocated to corporate quality work are insufficient • - The resources to help solve quality problems are scarce • - The implementation of process-oriented data quality management requires strong co-ordination and support (IT and statistics) • - Lack of standardisation of data treatment procedures and insufficient documentation of methods

14. Quality frameworks – in place • The main constraints on reaching optimal data quality in the statistical processes: • - The resources • - Difficult to apply the same quality concept (e.g. dimensions) for less developed statistical systems as for more developed systems • - The improvement of data quality is viewed as a gradual process that need to take into account resources constraints and establish priorities • - No direct contact with respondents • - The quality of the country data

15. Quality frameworks - not (yet) in place • The current quality work: • - implementation of new Statistical Information System (data validation/ consistency checks and metadata management) • - procedures for validation of statistics/ estimations by countries before publication • - regular review of data collection activities • - internal quality reports • - methodological aspects are documented and used to identify best practices • - project on “process documentation” to identify weaknesses and define best practices

16. Quality frameworks - not (yet) in place • Identification of weaknesses: • - manual data clearance procedures are well functioning but proves to be time consuming and intervenes at a late stage in report preparation • - data weaknesses are identified when in-house economists draft their analytical reports based on the statistics • - data weaknesses are identified on the basis of metadata available and by comparisons with similar statistics from other organisations • - the impact of data weaknesses is assessed using statistical “judgement”, improvement actions are discussed between statistician(s) and the supervisor and implemented according to agreed plans • - weaknesses in data are pointed out by the users (user feed-back or user surveys)

17. Quality frameworks - not (yet) in place • Main obstacles/ critical issues for implementation of quality assurance framework: • - economists and analysts perceive data quality requirements too excessive and seen as extreme refinements imposed by statisticians • - relevant IT tools should support the quality framework to efficient process of data and quality control (QF might be seen as a concept impossible to apply) • time and resources • A well-designed quality framework for organisations compiling statistics from different international organisations would ease the implementation

18. Quality frameworks – future plans (1) - review of existing quality framework - inform senior managers on data quality issues and principles governing international statistics - training of staff on quality issues and data processing - data collection by electronic means will increase and make it possible to implement more automatic data validation - SDMX is expected to be a mean for accessing data and metadata from countries and other international organisations - improved accessibility to data for users - adapt the common quality assurance framework that will be proposed by CCSA - establishment of a statistical data quality group

19. Quality frameworks – future plans (2) - promoting data and metadata dissemination - regular monitoring, assessment and reporting based on the statistical data quality framework - peer reviews of member countries

20. Quality reporting – the template • Section 6. Metadata about quality collected from data providers • - brief description of the quality reporting activties • - legal basis • - quality dimensions covered • - frequency • - procedures/ mechanisms in place for quailty reporting • - type of quality information (qualitative/ quantitative) • - the use of the quality information • - strengths and weaknesses of the quality reporting • - quality constraints • - quailty improvements • Section 7. Metadata about the quality of released statistics • - brief description of the ways of informing the users on the quality • - based on standardised reference metadata • - type of quality information disseminated (qualitative, quantitative) • - strengths and weaknesses of the system of informing users on quality • Section 8. For all international organisations • - the future plans for improving quality reporting • - additional comments/ recommendations etc. for this project

21. Quality reporting - results • “Legal” basis: • - Guidelines for good practices • - Ratification of Convention for data reporting and its implementation • - indirect through Council decision • - Standards • - Regulations • - Recommendation

22. Quality reporting - results • Coverage of quality dimensions (1): • - relevance, accuracy, credibility, timeliness, accessibility, interpretability, coherence • - relevance, accuracy, timeliness, punctuality, accessibility, clarity, comparbility, coherence, completeness, and sound metadata • - prerequisites of quality, assurances of integrity, methological soundness, accuracy and reliability, servicability, accessibility • - methodological soundness, accuracy, reliability, consistency, timeliness, and punctuality • - relevance, accuracy, timeliness, punctuality, accessibility, clarity, comparability, and coherence

23. Quality reporting - results • Coverage of quality dimensions (2): • - relevance, accuracy, interpretability, and coherence • - completeness, accuracy, reliability, comparability, adherence to standards • - data accuracy, comparability and consistency • - data consistency (aggregation) and comparability with standards • - methodological soundness (coverage) and conistency over time

24. Quality reporting - results • Frequency of the quality reporting: • - no common rules • - rolling programme • - once a year • - approx. every 5 years • - twice a year • - each time data submitted/ collected • - quarterly • - according to requirements in Regulations • - as part of projects on country assistance/ country reviews

25. Quality reporting - results • Procedures/ mechanisms in place for the quality reporting: • - the quality framework provides the theoretical and practical guidance • - metadata questionnaire attached to statistical data questionnaire • - manuals/ guidelines/ handbooks • - clerical and computerised edits • - standardised reporting forms for metadata • - glossary on quality • - bulletin board on data collection, dissmeination and quality of statistics • - standardised data quality assessment framework tool (part of formalised data quality progam) for assessing member countries data quality • - regulations • - metadata common vocabulary

26. Quality reporting - results • Type of quality information collected: • - almost all information provided are qualitative • - very few quantitative measures, like: • - response rates • - data completeness • - data timeliness • - revisions • ------------------------- • - binary indicators • - set of standard quality indicators • - overall assessments: • - allocation of “colour-codes” to country data • - quality ratings (four grade scale)

27. Quality reporting - results • Strengths: • - decentralised meaning that activity managers can collect the metadata • - metadata reporting by NSO’s are done in a systematic manner • - allows checking compliance against international standards • - a tool for monitoring where there are trade-offs between data quality problems and allocation of resources • - standard framework for metadata in general and metadata about quality in place covering many countries • - good cooperation with data providers • - quality criteria and quality reporting requirements included in legislations for member countries

28. Quality reporting - results • Weaknesses: • - un-coordinated • - too decentralised (no coordination) • - no standards/ standardised reporting • - no central monitoring • - needs further embedding in overall culture of the organisation • - reporting NSO’s not aware of international standard methodology, concepts, definitions and classifications • - no standardisation of metadata and rules to be followed • - metadata are not processed automatically • - infrequent collection of information • - burden on countries • - lack of resources • - self-assessments of countries not yet done • - not all statistics yet covered

29. Quality reporting - results • Coverage of quality constraints: • - none (9 out of 12) • - data sharing and coordination among data producing agencies • - confidentiality • - resources (financial, staff, technical) and efficient use • - planning of statistical programs • - covered by a special “merit and costs” procedure • - burden on respondents

30. Quality reporting - results • Coverage of quality improvement actions: • - no (8 out of 12) • - sometimes • - explicit sections on “plans for improvement” and “specific recommendations for improvement”

31. Quality reporting - results • Future plans for improving quality reporting from data providers: • - continued use of electronic dissemination • - investigation on the use of SDMX standards • - development of a metadatabase that allows automatic handling of metadata for the processing, tabulation and dissemination • - revision of the quality reporting framework in order to improve standardisation • - data quality assessment framework is being incorporated into the data dissemination standards • - development of a coherent quality framework with the aim to introduce a harmonised definition of quality • - workshops and seminars for countries on strengthening the capacity to produce data

32. Quality information for the users • Ways of informing the users on the quality: • - quality metadata is embedded in the dissemination system • - attached notes to datasets and tables • - metadata included in “Sources and Methods” publications • - standardised “country notes” on methods and definitions for each release • - Data Dissemination Standard(s) • - dissemination of data only possible if metadata are filled in • - annual quality report published summarising all economic statistics • - annual internal quality report for each statistical area • - quality profiles for key statistical indicators • - domain specific quality reports

33. Quality information for the users • Future plans for improving quality information about the published statistics: • - implementation of new Statistical Information System designed to fully support metadata (harmonisation and automatisation) • - more use of pre-coded response for metadata to improve standardisation • - adoption of systematic criteria is under consideration (dependent on international work) • - coverage of all statistical domains • - data quality requirements to be better integrated in the statistical database dissemination system