Census Evaluation

Census Evaluation Methodology and Practice 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Why evaluate ? Because the census is a huge operation (size, number of persons involved)prone to errors To provide users with some measures of quality of census data to help them interpret the results To identify types and sources of error in order to assist the planning of future censuses To serve as a basis for constructing a best estimate of census aggregates, such as total population, or to provide adjustment ofcensus results But…not to criticize the census takers !! 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

What errors in a census? • Coverage errors: • Errors in the count of persons or housing units resulting from cases having been “missed” or counted erroneously • Content errors: • Errors in the recorded characteristics of persons or housing units enumerated in the census (e.g. wrong age...) 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Coverage errors Omissions • Missing housing units, households and/or persons during census enumeration • If the whole housing unit is missed, all households and persons living in the housing unit will also be missed • Major causes of omissions are: failure to cover whole land area of a country in creating EAs; • Mistakes made by enumerators in canvassing assigned areas • Ambiguous definitions of EAs, unclear boundaries of EAs, faulty maps or coverage error during the pre-census listing exercise 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Coverage errors Omissions In addition, omissions within EAs can result because all or some of the members of the household were not present at the time of enumeration Proxy respondents can inadvertently or deliberately omit some members of a household 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Coverage errors Duplications • Occur when persons, households or housing units are counted more than once • Reasons for duplications include: • Overlapping of enumerator’s assignments owing to errors done during pre-census listing and delineation • Failure by enumerators to clearly identify boundaries • In practice, the number of omissions usually exceeds the number of duplicates (net under-counts) 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Coverage errors Erroneous Inclusions • This includes: • Housing units, households and persons enumerated while they should have not been(e.g. babies born after the census reference date) • Or enumerated in a wrong place 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Coverage errors Gross error • This is the sum of duplications, erroneous inclusions and omissions Net error • This is the difference between over-counts and under-counts • Net census under-count exists when number of omissions exceeds the number of duplicates and erroneous enumerations • Net census over-count is the converse 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Methods for census evaluation Non Matching Methods • Demographic analysis • Interpenetration studies • Comparison with administrative sources or existing surveys Matching Methods • Record checks • Post Enumeration Surveys 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Demographic Methods Example of methods: • Analyses of Age and Sex Distributions • Comparison of Successive Censuses • But also: • Stable Population Analysis of Age Distributions • Analysis Cohort Survival Rates 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Age and Sex Distributions Reasonableness of the age-sex distribution of enumerated population provides information on quality of the census Age-sex distribution for a given level of fertility, mortality and international migration follows a predictable pattern Unexplained departures from expected distributions signify existence of errors in census enumeration 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Age and Sex Distributions • Ratios and indices providing numeric departures of recorded age-sex distributions from expected distributions • Graphic presentation (population pyramid; cohort analysis) • Age ratios • Sex ratios • Summary indices • Whipples and Myers’ indices (digit preference/age heaping) • Age-sex accuracy index • Stable population theory (stable age distribution) • Quasi-stable population methods due to declining mortality • Internal consistency (e.g., age by marital status) 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Age heaping 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Comparison of Censuses • “Expected population” derived using population enumerated at previous census plus information on intercensal births, deaths and net international migration: • P1 = P0 + B – D + M + e(Population balancing equation) P1 = Population enumerated in census being evaluated P0 = Population enumerated in previous census B, D = Intercensual number of births and deaths M = Intercensal number of net international migrants e = residual • Requires accurate vital statistics and information on international migration • Or indirect estimates of intercensal fertility and mortality levels from a demographic survey can be used with previous census to derive an “expected” population • Need to adjust first census for net coverage error as well 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Demographic Analysis Advantage – no additional data is needed to be collected to perform the analysis Less costly In statistical offices with sufficient numbers of demographers there is no need for additional staff to do the technical analysis On the negative side these methods provide less insight into the different contributions of component errors to total error in the census Quality of sources (Vital Statistics…) 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Interpenetration Studies Method involves drawing subsamples, selected in an identical manner, from the census frame Each subsample should be capable of providing valid estimates of population parameters Assignment of personnel (i.e. enumerators, coders, data entry staff, etc.) is done randomly The method helps to provide an appraisal of the quality of census information and procedures 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Interpenetration Studies Gives good idea of different contribution of component errors to total census error Helps to identify operational stages that contribute to census error, thus identifying procedural limitations in a census Demerits include: That it is an expensive operation demanding many field staff, intensive training and close supervision Relatively complex in designing and implementation 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Comparison with existing sources • Surveys or persons or administrative sources can be used to evaluate coverage and content error in a census if: • They have identical items with same concepts and definitions • They are independent from the census • They have been conducted close to the census date 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Comparison with existing sources Review census results at aggregate rather than unit level i.e. provides only estimates of net census error Evaluates very limited characteristics such as sex and age distributions Merit They are relatively cheap compared to matching studies 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Matching Methods To evaluate content efficiently the following preconditions are essential: The record system should contain some relevant items covered in the census such as age, sex, education, relationship, marital status etc. Definitions of items should be identical between the census and the record system 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Record checks • Census records are matched with a sample of records from identification systems such as the vital registration system • The relevant respondents to the census questionnaire are traced to the time synchronized with the census • Sources include: • Previous census • Birth registrations • School enrolment • Citizen registration card • Immigration registers etc. 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Record checks Both coverage and content errors could be measured through the above comparisons To evaluate coverage efficiently the following preconditions are essential: A large proportion of census population should be covered in record system The census and record system should be independent from each other There should be sufficient information in records 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Record checks It provide separate estimates of coverage and content error Prospects of evaluating more characteristics compared to what can be done with non-matching studies Challenges Calls for high level technical skills including managerial Matching is expensive Countries that have used record checks include: Demark, Finland, Norway, Sweden, Taiwan and Canada 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Post Enumeration Survey (PES) • Consists of two separate collections: • A survey conducted using a sample frame independent of the census. • Persons from this survey are matched to the census to estimate the number of persons missed in the census • A survey conducted using a sample drawn from persons enumerated in the census. • This sample is re-enumerated to determine if the sample person or unit was erroneously enumerated (inc. erroneously located) 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Post Enumeration Survey (PES) Results can be used to evaluate the reliability of some characteristics such as sex, age, marital status, relationship to reference person or head of the household For some countries the results of PES can be used to adjust some census results Facilitates better interpretation of census results 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Planning of PES Preparatory activities Data collection related activities Matching Reconciliation (if undertaken) Data processing Estimation of coverage and content error Report preparation and dissemination 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

What is the P sample? The Population (P) sample consists a sample of clusters drawn from the same population but independent from the census Usually covers only household population Independently listed units that are valid and unique Results used to estimate census omissions 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

What is the E sample? The E sample is a sample drawn from cases already enumerated in the census Housing units enumerated in the census in the same sample area as the P-sample It is not uncommon to have an overlap of the E and P samples Results used to estimate census erroneous inclusions Estimate of erroneous inclusions provides a correction factor used in the Dual System Estimate of True Population 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Sampling Strategies Probability household survey It is usual to make inferences in a PES for a number of analytical domains Stratified cluster sample design is common First-stage units–area clusters/EAs PPS systematic sample selection Second-stage, usually canvass all households in selected EAs 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Why Area sampling? • At national level only a frame of EAs is required • Data collection is more efficient • Lower costs compared to simple random sampling (SRS) • Supervision is easier • However, estimates are prone to higher variability compared to SRS 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Sample Size Sample size depends on estimate requirements: • Geographic level (national, province, urban/rural) • Demographic (sex, age) • Reliability • Confidence level 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Some definitions Non-movers: persons who were in a particular household as of the census and PES date Out-movers: persons who were in a particular household at the census date but moved out or were not part of the household at the time of the PES In-movers: Persons who were enumerated in a particular household during the PES but where not there during the census date 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Matching 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Matching • Matching can be done by computer, manually or both • Basic process involves comparison of addresses, names & demographic characteristics • Housing units & persons enumerated in a census are compared with those enumerated in the PES • Computer matching can first involve scanning of questionnaire & installation of a matching software • Computer matching ensures speed and objectivity using matching algorithms 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Reconciliation Visits Helps to identify erroneous census enumeration Follow-up directed to no-matched persons and households Resolution of doubtful cases and definitive match status for every P and E sample element 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Dual System of Estimation (DSE) • Based on capture-recapture models • Chandrasekaran-Deming estimator (1949) assuming independence : 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Dual System of Estimation (DSE) True population 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Dual System of Estimation (DSE) Net coverage error • The difference between what should have been counted (True Population) and what was counted in the census Net Coverage rate • Net coverage error relative to the true population (an important indicator of the quality of census coverage) 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Dual System of Estimation (DSE) Gross coverage error The sum of Omissions and Erroneous Inclusions Gross coverage error rate per unit enumeration This is the absolute gross error relative to census enumerated population 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Post Enumeration Survey (PES) Merits: • Its results can be used to independently evaluate census coverage and content error, including reliability of selected characteristics collected in a census • Incorporates matching of individuals or units between the census and PES • Its results are generally more reliable than those of the census i.e. it justification for evaluation 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Post Enumeration Survey (PES) Challenges: Requires highly skilled field and professional staff Matching is complex As it is supposed to be carried out immediately after the census at times there is lack of adequate funds to implement the PES exercise 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

To go further Manuals: • Post Enumeration Surveys – Operational Guidelines. UNSD • http://unstats.un.org/unsd/demographic/sources/census/census3.htm 2010 World Population and Housing Census Programme Website: • http://unstats.un.org/unsd/demographic/sources/census/2010_PHC/default.htm 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Thank you 3rd Meeting of the Technical Coordination Group for the Population Censuses in South East Europe - Budapest

Census Evaluation