Evaluation of Census Quality and Coverage

1. Evaluation of Census Qualityand Coverage Preston Jay WaiteAssociate Director for Decennial CensusU.S. Census Bureau Presentation Before the Joint ECE/Eurostat Work Session on Population and Housing Censuses Geneva, Switzerland November 23, 2004

2. Summary of the REX Program: 1940-1990 • 1940 Census – included research and testing prior to and during the census • 1950 Census – initiated REX program; identified sources and measures of error and evaluated coverage • 1960 Census – included content and coverage evaluations and a post-enumeration survey • 1970 Census – included studies of content error, procedural evaluations, and an experiment to expand the mail census • 1980 Census – consisted of content, coverage improvement, and processing evaluations, experiments, and a post-enumeration survey • 1990 Census – similar to the 1980 program in content and scope

3. Overview of the Census 2000 TXE Program The Census 2000 TXE Program was the most ambitious formal test and evaluation program ever undertaken by the Census Bureau. • 87 Evaluations assessed the effectiveness of operations, systems, and processes, including the national paid advertising campaign, the automated data capture system which included OMR and OCR, address list development operations, coverage improvement operations, and a new multiple category option for race reporting. • 4 Experiments were conducted during Census 2000 because the decennial census environment provided the best conditions to learn about new methodologies. These included an alternative questionnaire experiment, an administrative records experiment, Social Security Number, attitudes, and privacy experiment, and a response mode and incentives experiment.

4. Overview of the Census 2000 TXE Program (cont’d) • Operational profiles documented number of cases, rates, data from production files, quality assurance, and information collected from enumerator or other staff debriefings. • Evaluations analyzed, interpreted, and synthesized the effectiveness of census components and their impact on data quality or census coverage. • We also conducted sample surveys, targeted reinterview surveys, and customer satisfaction surveys. • Some evaluations included focus group interviews to learn how or why respondents behave in a certain way. • We also used ethnographic and social network methods to study the effects of mobility on census coverage.

5. Coverage Measurement in the 1950-1990 Censuses • 1940 - An evaluation of the 1940 Census demonstrated the need for an assessment of census coverage. A match of draft registration records to the 1940 Census evaluated the coverage of adult males of an age eligible for the draft. Surprisingly, there were more males registered for the draft than enumerated in the census. • 1950 and 1960 - The Census Bureau conducted its first post-enumeration survey after the 1950 Census. The 1950 and 1960 post-enumeration surveys did not use dual system estimation to estimate census coverage error. Rather the idea was to form a much higher quality census interview with highly trained interviewers. The results of the post-enumeration survey interview were assumed to be the truth. • 1970 - The Census Bureau did not evaluate the 1970 Census using a post-enumeration survey. A decision was reached to use demographic analysis as the method for coverage evaluation in the 1970 Census. • 1980, 1990, and 2000 - implementations used dual system estimation.

6. The Census 2000 A.C.E. Survey • Litigation and controversy surrounded the use of a coverage measurement survey to adjust the census throughout the 1980s and 1990s. In January 1999, the U.S. Supreme Court ruled that the law covering the census, Title 13, did not permit sampling for apportionment of the House of Representatives. • The Census 2000 A.C.E. followed the basic methodology of post-enumeration surveys using dual system estimation.

7. A.C.E. and A.C.E. II Results • March 2001 - A.C.E. estimates became available in time to correct the Census 2000 redistricting files. However, on March 1, 2001, the Census Bureau released the "Report of the Executive Steering Committee for Accuracy and Coverage Evaluation Policy" which reported that we were unable to conclude, based on the information available at the time, that the adjusted Census 2000 data were more accurate for redistricting. Accordingly, we recommended that the unadjusted census data be released as the Census Bureau's official redistricting data. • October 2001 – After conducting further evaluations, the Census Bureau released the A.C.E. II report. After assessing considerable new evidence, we recommended that the unadjusted Census 2000 data also be used for non-redistricting purposes.

8. Revised A.C.E. II Results • March 2003 - The Census Bureau decided to produce intercensal population estimates based on the official Census 2000 results rather than on a population base adjusted using the revised estimates of census coverage. The decision was based on the results of further work to correct for detected errors in an effort now known as A.C.E. Revision II. • The Accuracy and Coverage Evaluation (A.C.E.) Revision II methodology represented a dramatic improvement from the previous March 2001 A.C.E. results. However, several technical concerns remained, including uncertainty about the adjustment for correlation bias, errors from synthetic estimation, and inconsistencies between demographic analysis estimates and the A.C.E. Revision II estimates of the coverage of children.

9. The 2010 Census Research Program – Differences from 2000 • The proposed 2010 Census program will make a distinction between true evaluations and “accounting type” assessments. • The Census Bureau executive staff plans to decide the high-level scope and focus of research programs before participating divisions develop research proposals. This will ensure that the selected evaluations and experiments address the most relevant topics and program funding is optimally expended. • Planning and design milestones for 2010 Census research components are slated to occur earlier than in the Census 2000 cycle.

10. The 2010 Census Research Program –Areas of Interest • The Master Trace Sample database was designed to help answer research questions that go beyond those addressed by the evaluations and experiments. The 2010 Master Trace Sample database should be expanded to include data on Group Quarters and coverage measurement data associated with persons. • We want to measure the effectiveness of procedures for unduplication, geocoding accuracy, the effectiveness of race reporting, the effect of residence rule strategies, and the impact of expanded use of automated tools by our field enumerators. • We intend to continue research on the use of administrative records to supplement development of the address list and to obtain data for nonresponse items. • We expect to continue improving upon efforts to measure census coverage, including net error as well as components of under- and over-enumeration.