240 likes | 381 Vues
Feel the Feed! InFuse and Dimensional Data for the UK Census and Beyond. Justin Hayes ESRC Census Dissemination Unit, Mimas Census 2011: Impact and Potential Exploring the Research Potential of the 2011 Census The University of Manchester 8 July 2011. Overview.
E N D
Feel the Feed!InFuse and Dimensional Data for the UK Census and Beyond Justin Hayes ESRC Census Dissemination Unit, Mimas Census 2011: Impact and Potential Exploring the Research Potential of the 2011 Census The University of Manchester 8 July 2011
Overview • Delivering the research potential • Straightening out the 2001 Census • Collaboration with ONS for 2011 • Benefits all round • What’s next? • Visioning a future data explorer • How to get on board • InFuse demonstration
Key Stakeholder Research • Data producers/providers • Data intermediaries/developers • End users
Delivering the research potential • Creating high quality content involves enormous effort and expense • Delivery is the last 100 yards of the census marathon • Potential remains just potential until the census is used, transforming it into impact
46,145 yards Image credits: Wolfgang Kumm / European Pressphoto Agency
Delivery requirements • Understand • Find • Use
Who to deliver to? • Everyone! • Censuses a key national resource • Make use easier for all to deliver best value • Encourage mass innovation • The coolest thing to do with your data will be thought of by someone else (Rufus Pollock) • Design with secondary use as a primary aim
Barriers to use of 2001 Census • Fragmented data • Inconsistent structures • Unnecessary complexity • Poor integration of metadata/meaning • Confusing disclosure control • Difficult to deliver • Difficult to understand, find, use
Age Banding in 2001 99 age bandings 76 unique to a single table
Straightening out the 2001 Census • Born of nine years of frustration followed by three years of hard work • Logical dimensional model based on SDMX Open Standard • Dissect and rationalise structures of original dataset to create new library • Integrate data and metadata using new structures
Delivery via a 2001 Census Data Feed • Structured, dimensional dataset • Logical model based on SDMX Open Standard • Dataset Description • SDMX and RDF Open Standards • Communication and transfer • RESTful Web service with API(s) • Publication for internal and external (soon) use • Suite of appropriate operations
Apps & Interfaces End Users Web Service & APIs Dataset Descriptions Developer Users
The InFuse interface • http://infuse.mimas.ac.uk • In service from beginning of May • Iterative design approach driven by user requirements • Table-free, lightweight, generic, modular • Operations on entire dataset • Currently academic use only
Collaboration with ONS for 2011 • 2001 Data Feed as feasibility study • Co-funding from ESRC and ONS to facilitate knowledge exchange • Assist with development of ONS 2011 API • Test data • Interface development • Richer dataset • Linked Data and development of 2001-2011 comparability
Benefits:Data Producers/Providers • Data Production • Management, control and authority • Integration of metadata/paradata(?) • Integration with other datasets • Exploit Mass innovation • Contact with user communities • Achieve strategic priorities • More efficient, effective and cheaper!
Benefits:Intermediaries/Developers • Lots of nicely structured and described data to mash • Easy automated/machine-to-machine operation • Generic, re-usable applications • Rapid development
Benefits:End User • Easier to understand, find, use • Purpose-specific, user-centred interfaces • More time for bigger, better, faster research
What’s next? • More and better data • Adoption of data feed approaches • Linked Data • Wider integration of datasets • Definitional • Geographical (GeoConvert web service) • Essential for B2011 in whatever form • Intelligent interfaces • Unforeseen innovation!
How to get on board: Producers/Providers • Produce structured data to Open Standards • Publish via APIs • Design for secondary use • Collaborate to promote and develop Open Standards • Cultivate developer interest
How to get on board:Intermediaries/Developers • Get tooled up! • Use Open Standards • Re-use and contribute code • Join and develop communities • Innovate to satisfy end user requirements • Lobby data producers/providers
How to get on board:End Users • Understand, find, use • Generate more impacts • Make your requirements known • Tinker