Using Performance Statistics to Design the 2006 Manufacturing Energy Consumption Survey

1. Using Performance Statistics to Design the 2006 Manufacturing Energy Consumption Survey April 6, 2006 Robert Adler (202-586-1134) Robert.Adler@eia.doe.gov Tom Lorenz (202-586-3442) Thomas.Lorenz@eia.doe.gov

2. Acknowledgements Census Bureau staff: Richard Hough Vicki Haitot Stacey Cole

3. Objective • To improve data quality in the Manufacturing Energy Consumption Survey by examining performance statistics on: • Data status flags • Differential response rates

4. Discussion Topics • Boosting participation in Internet electronic form • Other performance statistics that would be useful to track • Risks in targeting non-response follow-up to larger establishments

5. Organization • Background on MECS • Examination of data status flags to evaluate effectiveness of questionnaire • Comparison of status flag data: respondents who completed an Excel version vs a written version • Examination of differential response rate to suggest other changes to improve data quality in the MECS

6. Background MECS is a quadrennial survey of manufacturing industries based on the North American Industry Classification System (NAICS) • 3 survey forms • “A”: Mailed to most manufacturers • “B”: Petroleum refineries only • “C”: Sent to Chemical, Paper, Iron and Steel, and other NAICS industries that are energy intensive or have complicated energy flows • Self-administered written questionnaire

7. Background - continued • Designed and sponsored by EIA and conducted by the U.S. Census Bureau because data are confidential by Title 13 of the U.S. Code • Mandatory by federal law • Statistical sample from list frame (15,500 sample cases in 2002)

8. Data Status Flags R:Reported data; either through questionnaire or follow-up telephone or e-mail A:Analyst correction of reporting error I:Imputed data; missing data imputed by price or other information (rare) S: Source data: An entirely different set of data was used instead of MECS for the item E: System edit; data changed due to systematic edit because of common mistake in reporting

9. 3 2 1 Figure 1: Status Flags of Selected Data Items, Manufacturing Energy Consumption Survey, 2002

10. 3 2 1 Table 1: Type of Status Flag as Percentage of Total, Manufacturing Energy Consumption Survey, 2002

11. Results for Selected Items • Natural gas used as a fuel one of the most reported data items in the MECS • Important to minimize costly analyst intervention • The 22 percent “A” flags represent mostly conversions of the reporting units after failure of a price range check • Most of those analyst corrections could have been avoided if the respondents were warned of an edit failure while at the establishment site

12. Results for Selected Items - LPG Non-fuel • “S” flag for LPG nonfuel represents the substitution of Economic Census—Manufacturing (ECM) data for MECS • Data editors discovered that Chemical respondents did not interpret the MECS reporting requirements correctly • “Non-fuel” or “feedstock” use was not seen as a “material input,” the ECM term. Thus, as part of the 2006 MECS, we intend to use that term along side the other traditional MECS terms

13. Creation of Excel Version • EIA developed an Excel workbook version of the 2002 MECS C-Form • Excel version looked like the written questionnaire • Added capabilities to Excel version include: • price and value range checks of major energy sources • automatically calculates derived data items (e.g., total consumption of electricity) • automatically copies reported and derived data to later sections

14. Limitations Time and other constraints prevented it from optimal development • Not all desired onsite edits or screeners could be included because respondents had to save data to a CD and/or diskette • Data from the Excel version was not fully integrated with Census data capture routines • CD and/or diskette had to be mailed to the same Census address as the paper questionnaires • Responses were keyed the same as with the normal paper-and-pencil mode

15. Participation Rate • 323 responses from the Excel version (8.9% of the total number of C-form responders) • Excel responders accounted for 12.4% of the total consumption reported from the C-form • 23 respondents failed reporting using Excel version: • Incorrectly formatted diskette • Bad diskette

16. Electricity: Electronic vs. Paper

17. Natural Gas: Electronic vs. Paper

18. Differential Response Rates • As in the case of other establishment surveys, the problem of nonresponse in the MECS has been growing. Possible causes: • Downsizing and greater emphasis on “core functions” • Attitude toward surveys • On the other hand, the budgets necessary to conduct large government surveys are also being squeezed • To best use the resources available to us, the non-response follow-up for the 2002 MECS was targeted to have the greatest effect on the survey data quality

19. Table 4: Response and Coverage Rates by Cell Size and Type.

20. Conclusions and 2006 MECS • 2006 MECS will shift to an Internet-based electronic reporting method for its main mode of collection • Much of the 2002 MECS late- and non-response was due to not having the right contact persons at the establishment in the beginning • Status flags will be amended to further define the occurrence and type of analyst intervention

21. Questions for the Committee • Given that only 10 percent of the eligible respondents chose to use the electronic Excel form in 2002, have we introduced bias into the comparisons? Are we still justified in reporting the results? • How do we boost participation in the Internet electronic form, given that respondents will have a back-up available to them?

22. Questions for the Committee - continued • What other performance statistics would be the most useful to track? • What are the risks associated with targeting non-response follow-up to larger establishments to minimize a non-response adjustment?