NEW DIRECTIONS in DATA COLLECTION and ANALYSIS

1. NEW DIRECTIONS in DATA COLLECTION and ANALYSIS The Role of Official Statistics 2014 IAOS Conference Da Nang, Vietnam Hermann Habermann

2. “It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, it was the epoch of belief, it was the epoch of incredulity. . .”

3. Current Model - Strengths Survey Sampling Is a Well-Established Scientific Method • Probability Based/Generalizable • Error Structure • Time Series

4. Current Model - Strengths (Cont.) • Founded on Idea that Information is Owned by Individual/Permission Needed • Publicly Developed Rules for Privacy and Confidentiality • Accepted as Gold Standard • Enabled International Comparisons

5. Current Model - Weaknesses • Sampling is a Mature Technology • Seminal paper on Theory of Estimation Based on Confidence Intervals by Neyman in 1934 • Turning Theory into Practice by Hanson and Hurvitz in 1943 – multi-stage sampling • Government Budgets Are Under Attack • Official statistics are often viewed as too costly • Not timely • Difficult to recruit

6. Current Model – Weaknesses (Cont.) • Administrative Records – Public and Private Are Appealing • Census Becoming Increasingly Problematic • Production of Small Area Data Costly and Complex • Response Rates Are Falling

7. Falling Response Rates

8. Summary of Current Model • While Still the Gold Standard • There Are Continuing Demands for • Data collection that is cheaper and more timely • Data analysis that is more relevant and at smaller areas • Transformative Movements Exist

9. Three Transformative Movements Are Changing the Future • Control of Data by Corporations • Rising Influence of NGOs • Population Movement Toward Cities

10. Control of Data by Corporations • Corporations Always Interested in Collecting and Analyzing Data about Individuals • Were Limited by Computer Power, Storage Costs, and Analytic Techniques • Technological Change • Allows for greater data collection • Greatly reduces the cost of storage • Increases modalities of data collection • Wearables, smart phone integration, use of nanotechnology

11. Examples • Reduction in Storage Cost Perhaps Most Important • $450,000 per gigabyte in 1980 • $.05 in 2013 • If air travel improved at same rate since 1980, a 16 hour flight would now take 5 milliseconds • Google Corporation Estimated to Store About 10 E or One Million Times the Text Stored at the US Library of Congress

12. Examples (Cont.) • Collecting, Aggregating, and Brokering Personal Data • Second Largest US Company in this Field, Acxiom • 23,000 computer servers • 50 trillion data transactions per year • 1,500 pieces of data per consumer • Data from Publicly Available Records • Online Behavior Tracked by Cookies, Browser Advertising • Survey Data

13. Examples (Cont.) • Over 1.2 billion Facebook Monthly Active Users1 • Facebook Makes Its Own Rules • Recent manipulation • Weibo Has over 500 Million Users http://www.statista.com

14. Second Transformative MovementRising Influence of NGOs • NGOs Still Controversial but Significant in Large Parts of the World • NGOs Are More Nimble – Do Not Have to Satisfy a Government Constituency • More Easily Integrate Data Estimates and Analysis • Computing Power and Software Tools Available

15. Alternative Producers of Official Statistics? • Pew Foundation • Gates Foundation - Financial Inclusion Tracker . Surveys in Uganda, Tanzania and Pakistan • Monster.com/Google/MIT - Billion $ Price Project • ADP Employment Estimates

16. Alternative Producers of Official Statistics? (Cont.) • Alternatives Are • Cheaper • More timely • Small area data • Deemed “Fit for Use”   • Are Official Statistics Still Official?

17. Third Transformative MovementPopulation Shifts to Cities • In 1960 34% of Population Lived in Cities • In 2014 it was 54% and • In 2050 projected to be 66% UNFPA/Forbes

18. Why Concentrate on Cities • Policy Officials Focused on Building Smart Cities • Singapore/India project • Engines of Economic and Social Growth • Have the Critical Density to Facilitate the Investments in Multiple Sensors

19. New Cities Will Be Smart • Creating Transport Systems to Avoid Congestion • Construction Using Low Energy Housing Materials • Extensive Use of IT Systems to Run Various Urban Functions and the Provision of E-Government Services to its Citizens

20. Example City Projects • New York City uses data to remedy a wide range of issues affecting short and long term issues — from commuting times to emergency preparedness to air quality. • City Projects Bring Together Computer Scientists, Social Scientists, Modelers, and Statisticians Center for Urban Science and Progress cusp.nyu.edu

21. Research Directions • Developing the Science of Big Data –Integrate Mobile Technology, Embedded Sensors, Blogs, Social Media, and Location-Based Tools, with Traditional Sources of Data • Future Cities Laboratory Focuses on Sustainable Urbanisation The Social and Decision Analytics Laboratory at Virginia Tech, Singapore-ETH Zurich Center

22. Challenges for NSOs • NSOs Are National, Corporations International • Working with Users on “Fitness for Use” • Integrating Corporate and NGO data • Incorporating Terabytes of City Data • Integrating Computer Scientists, Social Scientists, and Statisticians

23. Initiatives • Examples • Indonesia Using Twitter to predict commodity prices • Australia looking for evidence of the use of tax havens, and data-matching to identify small online retailers that are not meeting their compliance obligations • Universities building consortiums to develop big data science • Use of commercial credit card data in GDP • Estimation using unstructured data e.g. Facebook • International Meeting Programs Address Many of These Issues

24. Challenges for International Agencies and Societies • Creating a Neutral Big Tent • Minimizing the Gap between Developed and Developing Countries • Engaging With Existing Efforts to Define Privacy/Confidentiality • Supporting a Unifying Approach

25. Suggestions for IAOS • Support and Disseminate Good Practices • Develop Programmes for UN Statistical Commission • Insure Representation at International Activities • Develop privacy and confidentiality policies • Develop a Continuing Presence • Create a semi permanent position

26. “It is a far, far better thing that I do, than I have ever done…”