Data collection and processing, data quality

1. 10 Data collection and processing, data quality Heinrich Brüngger Management seminar on global assessment Yalta, 23 - 25 September 2009 Session 10

2. Part I: Data collection and processing • Part II: Confidentiality (with emphasis on data collection and processing) • Part III: Data quality (quality management)

3. Nature of decisions on methods of data collection and processing • Under professional independence (FP 2), therefore within the statistical system without outside interference once the decision about the “what” to measure in broad terms have been made • Decisions about what official statistics should measure may involve the political level, and have to involve discussions with a broad range of users

4. Professional independence • The “how” part of statistical activities has to focus exclusively at obtaining the most reliable representation of the phenomenon of the real world with the given resources • Scientific methods, international standards, and empirically established good practices are the best guide for professional independence • The statistical law may require the adoption by government in the case of statistical surveys because of explicit legitimacy in the interaction with respondents and the burden caused to respondents, but governments should not interfere in methodological issues

5. Main data sources (FP 5) • Official statistics has to have the right to use three types of sources: • Statistical surveys: data collection from individuals, households, corporate and unicorporated businesses, and public entities outside the own government structure, for the exclusive purpose of statistics (most of which official statistics) • Administrative records: secondary use of data collected primarily for administrative purposes about individuals or private businesses by other parts of the same government structure

6. Main data sources (ctd.) • Secondary use of environment and territorial observation and monitoring data collected by specialised government agencies • For statistical surveys, the statistical law applies to all phases of the statistical activity; for the other two sources, primary data collection has to be based on other laws, and the statistical legislation becomes only applicable once data are handed over to a producer of official statistics

7. Statistical programmes • If the law requires input-oriented programmes, a clear distinction has to be made between statistical surveys, where the statistical law and the statistical programmes are the legal basis for data collection, and the second and third data source, where the primary data collection is for another purpose and has therefore to have a different legal basis

8. Minor data sources for national official statistics • Official statistics from a partner country or from an international organisation (especially for international flows) • Registers in the hand of specialised private companies (as one of the sources used to keep statistical registers updated)

9. Statistical surveys • Exhaustive vs. sample surveys (mixed forms possible) • Mail, face to face interview, telephone interviews, internet (mixed forms possible) • Compulsory participation of the respondent vs. right to decline participation (mixed forms possible) • Recall and follow-up policy for non-response, depending on type of survey

10. Response burden • Response burden has to be assessed in advance and taken into account: • all new surveys have to be tested first (pilots) • all respondents have to be informed about the purpose and legal basis of the survey, and especially about the confidentiality measures • forms and questions have to be understandable, non-intrusive, and make answers possible for him/her from memory or from existing material (businesses)

11. Response burden (ctd.) • first reminders have to be proportionate • response rates have to be closely monitored The NSI has also to act as custodian for the respondents; users will tend to minimise the issue of response burden

12. Surveys (ctd.) • Information given by a respondent in statistical surveys have no legal implications for him; they can (and should) therefore be checked and when necessary altered (edited) by the NSI • In many cases, the characteristics of interest to users cannot be taken over directly into questionnaires, but have to be compiled as derived characteristics using several questions

13. Non-response (valid for all data sources) • The NSI has to develop a policy for treating general and item non-response, which has to be subsequently fine-tuned and documented for each survey • Imputations for missing (or unusable) information is legitimate, but a careful balance has to be found between “not enough” and “too much” (risk of introducing biases)

14. Administrative sources • The statistical office must have the legal and de facto right to receive, for its tasks, regularly and on an ad hoc basis, microdata sets from other ministries and public entities taken from their administrative sources • This does not mean that direct identifiers have to be included in all cases, but the possibility should not be entirely excluded

15. Administrative sources (ctd.) • Statistical producers have the right to alter (edit an impute) the administrative data received from other ministries to improve compatibility with statistical definitions and classifications • Data received in this way should never be given back to the data owner or transferred by the statistical producer to a third party for administrative purposes

16. Relationship between data sources • Before considering introducing a new survey, or expanding an existing one, for new information needs, other possible data sources (especially administrative) have to be examined for their potential • The possibility of combining questions about different areas in the same survey is limited: the survey system has to be structured carefully

17. Relationship between data sources (ctd.) • Administrative sources are in constant change, and some risk of disappearing completely • The final decision-making about administrative sources is outside the statistical system. Therefore, the NSI has the responsibility to constantly monitor (if possible anticipate) and react to planned changes that may affect the use of administrative sources for official statistics positively or negatively

18. Relationship between data sources (ctd.) • NSIs have to avoid considering data sources, and especially surveys, as stove-pipe type of parallel operations, but rather as a system of interrelated operations with n:m relationship between data sources and results • Very often, the best estimates are based on a judicious combination of sources, combining the individual strengths and reducing the individual weaknesses • Statistical registers of good quality are a cornerstone for a good survey system, and for matching different sources

19. Relationship between data sources (ctd.) • When results on the same issue are published from different sources separately (i.e. without prior integration), an accounting relationship between the results should be established, with the components quantified if possible • Otherwise, only the most complete aggregate should be released as results of official statistics in absolute terms

20. Confidentiality (FP 6) • Statistical confidentiality aims at protecting the privacy of individual units about which data are collected and processed. It has two components: • Producers of official statistics do not disclose, either directly or indirectly, characteristics about protected units to any third party in such a way that any user might derive additional information about a protected unit

21. FP 6 (ctd.) • Producers of official statistics use data about protected individual units only for statistical purposes (official statistics in the first place) • The statistical system, and in the first instance the NSI, have to make sure in an effective a credible way that the two elements of the confidentiality principle are strictly followed in all parts of the production and dissemination processes, and that confidential data are stored in a safe way

22. Confidentiality in data collection • Legal situation in countries may differ with respect to: • Protected units: physical persons, private households, private businesses (whether legal persons or unincorporated enterprises) should be fully protected; government units as institutions cannot invoke statistical confidentiality; public enterprises in a competitive market are a borderline case • All characteristics treated alike, or some considered to be free (e.g. for legal persons), or particularly sensitive (higher degree of protection, e.g. about health, crime, ethnicity)

23. Confidentiality in data collection (ctd.) • Data collection in statistical surveys: replies of respondents should go back as directly as possible to the statistical producer. No other government agency should be involved as intermediary (regional level!) • Data processing: except in the context of statistical registers, identifiers of units should be separated from the context variables by the statistical producer at an as early stage as possible, and either stored separately or destroyed

24. Confidentiality in data collection (ctd.) • Data security: final (edited) microdata sets have to be stored safely, and access limited to those within the NSO who use the data regularly. No direct access from outside the NSO should be possible. For files with identifiers (other than statistical registers), each access has to be documented. • Data security, update and access regulations for statistical registers have to be worked out carefully in a separate book of rules.

25. Quality management: transparency as prerequisite • All decision on methods (collection/processing/dissemination) have to be documented and made accessible for the public (FP 3) • Quality management is addressed by several principles of the EU Code of Practice • Quality management refers to products, processes and staff

26. Quality management: products • Products to be regularly assessed with pre-defined quality criteria (and rejected if minimal quality level for fitness of use not met): • Final micro-data sets from a data collection instance • Results of official statistics prior to release • Products to be assessed occasionally • Questionnaires for surveys • Dissemination products

27. Products to be released • An important element of professional independence is the right of the NSI not to release official results if pre-defined minimal quality criteria are not met; this has to be a decision by the chief statistician • This does not preclude the use and release of such results as one input among others into a broader (e.g. geographical) aggregate

28. Quality management: processes • Processes should be defined in terms of who decides on what and when within the statistical system, and who has to be consulted and informed; they have to be documented for mainly internal use • Processes should be defined for each phase in a generic statistical business model (from design to archiving)

29. Processes (ctd.) • It is important to define the system of processes and responsibilities in such a way that the tendency for creating “black boxes” by responsible organisational units is offset. Stove-pipes (same organisational unit responsible for all phases of the production process for a given activity such as survey) are problematic also from this point of view

30. Processes (ctd.) • The process definitions will be different between on-going activities (with room for small improvements), and major redesigns/innovations, and between production activities ending in the release of results of official statistics and infrastructure activities such as dissemination or training

31. Cross-cutting evaluations • An overall evaluation of the portfolio of activities and the extent to which the needs of users are met is necessary when the preparation for the next multi-annual programme starts • Extensive user consultations are necessary, and external expertise (such as in a GA) may help in identifying and prioritising gaps to be addressed in the next programme

32. Tendency in organisational structures of NSI • First-level division by broad subject areas used to be predominant • Recent shift to functional break-down at first level (data collection; processing; dissemination) for various reasons, among others improved quality management • Senior management has to provide incentives for quality management leading to measurable improvements

33. Staff management • Quality management has to become an almost automatic part of the activity for each staff member • Regular training on management issues should include a substantial part on quality management • Regular assessment of the individual performance of each staff member can be very instrumental, if not implemented as a bureaucratic routine, but as open dialogue towards improvements

34. Thank you for your attention