Principles, criteria and methods Part 1

1. Quality management Principles, criteria and methods Part 1 Produced in Collaboration between World Bank Institute and the Development Data Group (DECDG)

2. Definitions of quality/I • Quality may be related either to subjective feelings or to objective facts • Objectively, the quality of something is the sum of its essential attributes or properties. • Quality has to do with user needs and satisfaction. • Users can not define the quality standards. • A minimum standard is to be transparent about the way in which concepts and methods are used.

3. Definitions of quality/II • Subjectively, something has quality when it meets the expectations of the user • The International Organization for Standardization (ISO 8402) defines quality as: ‘Quality is the totality of features or characteristics of a product or service that bear on its ability to satisfy stated or implied needs of customers’. • Openness about the data shortcomings and needed data revisions can help in getting understanding and support.

4. Quality frameworks • Some international organizations have developed quality frameworks for official statistics • These frameworks generally use the same kind of criteria, but there are slight differences • One of the most comprehensive quality framework for statistics is the Data Quality Assessment Framework (DQAF), developed by the IMF • Another useful international framework is the European Statistics Code of Practice (ESCP)

5. DQAF Quality dimensions • The DQAF uses the following main dimensions of quality: • Prerequisites of quality • Assurances of integrity • Methodological soundness • Accuracy and reliability • Serviceability • Accessibility

6. Prerequisites of quality • Prerequisites are not quality criteria, but refer to institutional quality elements: • Legal and institutional environment - the environment is supportive of statistics • Resources - resources are commensurate with needs of statistical programs • Relevance - statistics cover relevant information on the subject field • Quality management - quality is a cornerstone of statistical work

7. Prerequisites – the indicators/1 • The responsibility for collecting, processing, and disseminating the statistics is clearly specified • Data sharing and coordination among data producing agencies are adequate • Individual reporters’ data are to be kept confidential and used for statistical purposes only • Statistical reporting is ensured through legal mandate and/or measures to encourage response • Staff, facilities, computing resources, and financing are commensurate with statistical programs • Measures to ensure efficient use of resources are implemented

8. Prerequisites – the indicators/2 • The relevance and practical utility of existing statistics in meeting users’ needs are monitored and reported publicly • Processes are in place to focus on quality • Processes are in place to monitor the quality of the statistical program and reported publicly • Processes are in place to deal with quality considerations in planning the statistical program and reported publicly

9. Assurances of integrity • Explanation: the principle of objectivity in the collection, processing, and dissemination of statistics is firmly adhered to • Elements are: • Professionalism • Transparency • Ethical standards

10. Integrity – indicators/1 • Statistics are produced on an impartial basis • Choices of sources and statistical techniques as well as decisions about dissemination are informed solely by statistical considerations • The appropriate statistical entity is entitled to comment on erroneous interpretation and misuse of statistics • The terms and conditions under which statistics are collected, processed, and disseminated are available to the public

11. Integrity – indicators/2 • Internal governmental access to statistics prior to their release is publicly identified • Products of statistical agencies/units are clearly identified as such • Advanced notice is given of major changes in methodology, source data, and statistical techniques • Guidelines for staff behavior are in place and are well known to the staff

12. Methodological soundness • Explanation: The methodological basis for the statistics follows internationally accepted standards, guidelines, or good practices Elements: • Concepts and definitions have to be in accord with internationally accepted statistical frameworks • Scope must be in accord with internationally accepted standards, guidelines, or good practices • Classification/sectorization are in accord with internationally accepted standards, guidelines, or good practices • Basis for recording – Flows and stocks are valued and recorded according to internationally accepted standards, guidelines, or good practices

13. Methodology – Indicators/1 • The overall approach to statistics, in terms of concepts and definitions follows internationally accepted standards, guidelines, or good practices, as well as the scope, and the classification system • The scope is broadly consistent with internationally accepted standards, guidelines, or good practices • Classification systems used are broadly consistent with internationally accepted standards, guidelines, or good practices

14. Methodology – Indicators/2 • Market prices are used to value flows and stocks • Recording is done on an accrual basis • Grossing/netting procedures are broadly consistent with internationally accepted standards, guidelines, or good practices

15. Accuracy and reliability/1 • Explanation: Source data and statistical techniques are sound and statistical outputs sufficiently portray reality Elements: • Source data available provide an adequate basis to compile statistics • Assessment of source data • Statistical techniques employed conform to sound statistical procedures • Assessment and validation of intermediate data and statistical outputs - • Correct application of data collection, processing and data analysis. • Correct sampling • Correct application of the mathematical tools.

16. Accuracy and reliability/2 • Revision studies - Revisions, as a gauge of reliability, are tracked and mined for the information they may provide • New methods may be needed to be introduced and existing methods may be needed to be revised and changed. • Revisions also influence back-data.

17. Accuracy and reliability – Indicators/1 Indicators of accuracy and reliability include several aspects related to source date: • Data collection programs take into account country-specific conditions • Source data reasonably approximate the definitions, scope, classifications, valuation, and time of recording required • Timeliness • Routinely assessment • Data compilation employs sound statistical techniques to deal with data sources • Other statistical procedures (e.g. data adjustments and transformations) employ sound statistical techniques

18. Accuracy and reliability – Indicators / 2 • Intermediate results are validated against other information where applicable • Statistical discrepancies in intermediate data are assessed and investigated • Statistical discrepancies and other potential indicators or problems in statistical outputs are investigated • Studies and analyses of revisions are carried out routinely and used internally to inform statistical processes

19. Serviceability/1 • Explanation: Statistics, with adequate periodicity and timeliness, are consistent and follow a predictable revisions policy Elements: • Periodicity and timeliness - Periodicity and timeliness follow internationally accepted dissemination standards

20. Serviceability/2 • Consistency - Statistics are consistent within the dataset, over time, and with major datasets • Revision policy and practice - Data revisions follow a regular and publicized procedure

21. Serviceability - Indicators • Periodicity and timeliness follows dissemination standards • Statistics are consistent within the dataset • Statistics are consistent or reconcilable over a reasonable period of time • Statistics are consistent or reconcilable with those obtained through other data sources • Revisions follow a regular and transparent schedule • Preliminary and/or revised data are clearly identified • Studies and analyses of revisions are made public

22. Accessibility • Explanation: Data and metadata are easily available and assistance to users is adequate • Elements: • Data accessibility - Statistics are presented in a clear and understandable manner, forms of dissemination are adequate, and statistics are made available on an impartial basis • Metadata accessibility - Up-to-date and pertinent metadata are made available • Assistance to users - Prompt and knowledgeable support service is available

23. Accessibility – Indicators/1 • Statistics are presented in a way that facilitates proper interpretation and meaningful comparisons (layout and clarity of text, tables, and charts) • Dissemination media and format are adequate • Statistics are released on a pre-announced schedule • Statistics are made available to all users at the same time • Statistics not routinely disseminated are made available upon request

24. Accessibility – Indicators/2 • Documentation on concepts, scope, classifications, basis of recording, data sources, and statistical techniques is available, and differences from internationally accepted standards, guidelines, or good practices are annotated • Levels of detail are adapted to the needs of the intended audience • Contact points for each subject field are publicized • Catalogs of publications, documents, and other services, including information on any changes, are widely available