CEMS: The Facility for Climate and Environmental Monitoring from Space

1. CEMS: The Facility for Climate and Environmental Monitoring from Space Victoria Bennett, ISIC/CEDA/NCEO RAL Space

2. CEMS– what is it • A joint academic-industrial facility for climate and environmental data services • Based in ISIC, Harwell • Alongside Visualisation Centre, EO Hub, SRU

3. CEMS– what is it CEMS will provide: • Access to large-volume climate and EO datasets, alongside processing capability; • Commercial and scientific applications and services, hosted alongside key datasets; • Data quality, integrity and visualisation tools alongside advice and consultancy; Initial partners:

4. CEMS Overview

5. CEMS Infrastructure • Layered architecture • High performance storage (1.7 PB) • Panasas parallel filesystem: resilience, scalability, fast performance, eliminating I/O bottlenecks • Processing hardware (20 nodes, each with 12 cores) • Managed through cloud-based environment • Virtualisation layer based on VMware • Allows computing resources to be used by third parties via a cloud • Platform to host applications and services for academic and commercial user communities • Dual site • Deployed across neighbouring academic and commercial sites on the Harwell campus

6. Data Harwell Site, Oxfordshire Initial CEMS funding has enabled the creation of new state of the art infrastructure to host data, and services to process it EO datasets are being transferred from UK’s NEODC (NERC EO Data Centre) to CEMS infrastructure (a huge undertaking) New datasets to follow CEMSalso integrates with other UK academic activities • JASMIN: access to CMIP5 for intercomparison with model data • STFC Scientific Computing: SCARF processing cluster and tape back-up Electron Building, ISIC R89 Building, RAL STFC STFC SCARF HPC (2000 CPUs) ATLAS Tape Store (3.5PB) CEMS Commercial CEMS Academic JASMIN 10GBit link ISIC - STFC 10 x compute nodes 10 x compute nodes 20 x compute nodes 0.7PB Storage 1.1PB Storage 3.5PB Storage 1100 blades, fast storage connected into low latency network Data sharing and integration of services between academic and commercial sites

7. CEMS Concept – multi-level approach

8. Exploiting Cloud Computing for CEMS • CEMS provides a flexible resource for processing and storage of EO data, using acloud computing model • Different groups can be allocated portions of the storage, network and compute infrastructure • tailored to their needs • without the need for the upfront capital for hardware purchase. • We’ve built links with the user community, including NCEO and the ESA CCI programme • First users are accessing the system

9. Core System • The core system provides the underlying functionality for CEMS to operate • It builds on top of hardware and cloud virtualisation layers • Key use cases were prioritised • User management and support services • Data discovery and access • Virtual machine access • This is a baseline upon which to build

10. Applications and Services • Demonstrator applications were developed in parallel with the deployment of the infrastructure • These proto-CEMS apps illustrate and showcase the possibilities for use of CEMS • Interactive (web based) applications • EO Data Processing • Aimed to demonstrate one of more of the following: • Complex or demanding processing of large datasets • Bringing multiple datasets together • Visualisation ISIC video wall showing Model Data (Met Office Hadley Centre HADGEM2 model, near-surface air temperature) and NASA Blue Marble on interactive spinning globes OceanDIVA demonstrator developed by Reading e-Science Centre with data and scientific advice provided by University of Edinburgh

11. Applications and Services: OceanDIVA • Quick-look tool for comparing model/assimilation with in-situ data

12. CEMS Evolution CEMS Components Engagement Timeline • Identifying the goals • Establishing initial collaboration Sept 2012+: 1st cloud users, growing user community Applications Support services, consultancy Core System Aug 2012: core software services +virtualisation • Service providers build apps to host on CEMS • First hosted processing for research groups and industry Technical implementation Virtualisation and Cloud Mar 2012: H/W delivery + demonstration S/W Dec 2011: H/W purchase • Est. CEMS Board and Integrated Project Team, groups: e.g. outreach, business modelling • An eco-system of applications and support services for • End user communities Hardware Demonstrators Data Integrity Nov 2011: £3m funding from UKSA CEMS Vision August 2011: User Requirements Analysis

13. Who will use CEMS, and how? • Bids/Proposals – CEMS usage in projects (ESA, EC, UK) • Private Use – use of CEMS to support internal use e.g. data, compute or core services to support product development • Application Hosting – infrastructure for hosting and deployment, as well as a shop window to reach a large customer base. • Academic Usage – scientific community processing and storage of long time series of data • Community Data Hosting – we are looking to build up the data offering • Capability Development – tools or core services (e.g. to manipulate, inter-compare or visualise data sets) developed and supplied by users/customers

14. Current Status and Future Developments • CEMS is open for business: • http://isic-space.com/cems/ • Contact cemsinfo@isic-space.com • Future developments of the system: • Storage and hardware: expand to support user community • Cloud: collaboration with external partners – GSCB, European Grid Infrastructure, Open Source clouds • Core system: Earth System Grid Federation – a federated infrastructure for climate data, suitable for EO also • Apps and services: • Hosted processing – OGC WPS • Hosted interactive development environments – iPython Notebook • Individual user exploitation of cloud • IPython Notebook interactive environment • Virtualisation

15. Questions