NERC Environmental Big Data Capital Bid

1. NERC Environmental Big Data Capital Bid Neil Lonie & Andrew Brooks

2. Dundee Satellite Receiving Station (DSRS) Overview Based in Space Tech. Centre, Univ. of Dundee – Director Prof Steve Parkes Operational since 1976 – Part of NERC Services & Facilities group – Initially funded via grant then contract since 1978 Worked in partnership with Plymouth Marine Lab. Remote Sensing Group for many years – Also NERC S&F funded Formalised this in 2006 – Established NERC Earth Observation Data Acquisition and Analysis Service (NEODAAS) DSRS primary activities – direct satellite data reception, archiving and distribution to support UK environmental research Receive from various EO satellites – primarily polar orbiting with Direct Broadcast capability, e.g. NOAA/MetOP-AVHRR and Terra/Aqua-MODIS Currently receive around 30 overpasses per day – Archive of over 160,000 passes back to 1978 Also receive geostationary satellite data giving full global coverage Operate 5 tracking antennas for polar and 2 fixed antennas for geostationary data reception

3. Raw received data are processed to and distributed in standard low-level formats, e.g. NOAA Level 1b Imagery extracted from raw data are also provided in standard image formats and map projections Higher-level processing and data analysis generally undertaken by NEODAAS facility partners at PML or users NEODAAS-Plymouth provides products derived from Dundee data plus NASA and ESA global data sets Products include Sea-surface temperature, ocean colour, front maps, time composites Supplied in near-real time or from archive NEODAAS user community extends across most NERC science areas – e.g. Marine, Atmospheric, Earth, Terrestrial, Polar DSRS Overview (contd.)

4. DSRS Facilities

5. What is the asset? Our bid is for a computer hardware asset comprising three main elements: 8 Data Processing server computers 2 Web server computers 2 Uninterruptible Power Supply (UPS) units Data Processing servers Will significantly enhance our data processing capability Enable more efficient processing of raw data we receive to user products Provide resources to increase range of products we can make available to users For both new data in near-real time and our archive data Web servers Will replace ageing servers used at present Provide user access to and distribution of products Improve capability to support wider range of products, increased demand and more users Provide resources for longer-term redevelopment of website UPS units Ensure stable mains power supply to the servers Protect against mains power glitches and longer outages

6. How asset will be used and by whom Asset to be integrated into our data processing and dissemination infrastructure Processing servers will have current and new software installed They will convert raw data we receive to Standard low-level formats Image products New higher-level products facilitated by NASA, EUMETSAT etc. processing software In near-real time a.s.a.p. after reception Also create new products from our archive data Resulting products to be held on our online storage Web servers to be used as primary method of dissemination Configured to replace old systems for user access to data and products Website developed to allow new products to be accessed Also allow full redevelopment of our website for launch later

7. How asset will be used and by whom (contd.) 10GbE Network Switch Data processing system Antenna control & reception systems Raw data L0, L1 & Products Web/FTP servers Data & Products Level 0 data 140Tb Data Storage Array NEODAAS Plymouth Level 1 data New Products Image Products Raw received data Level 1-4 data Marine Products DATA USERS LTO Tape Archives

8. How asset will be used and by whom (contd.) DSRS will be primary user – generating data and products and making these accessible Our users will be the beneficiaries – improved access and a wider range of products Users will include NEODAAS partners at PML – to derive high-level Marine products in particular from our data (e.g. SST, ocean colour) Marine scientists through product access above via PML Atmospheric scientists through access to products via our website Terrestrial, Earth and other science areas – new products using software designed for DB groundstations International researchers, Education/Training and Public through direct access to products via our website

9. Why the resource is needed DSRS receives data directly from polar orbiting EO satellites Required to process raw data and provide useable data and images a.s.a.p. after reception Rapid access is essential for NRT requirements – Research cruises and Aircraft campaigns E.g. cruise ship and FAAM aircraft supported during two campaigns of NERC ACCACIA programme this year Started receiving from new MetOp and Suomi-NPP satellites in 2012 These yield higher volumes of data and significantly increase processing times – 1-2 hours rather than minutes for older satellites E.g. new NPP data can be received before previous overpass is processed The resource will enable faster processing of data we receive – Data and products available to users much faster Provides service continuity users require

10. Why the resource is needed (contd.) DSRS processes data to standard low-level formats and viewable imagery High-level processing carried out by our NEODAAS partners (primarily Marine applications) and users in some cases – Requires scientific input/expertise Processing software from agencies such as NASA and EUMETSAT is Intended for use by DB groundstations such as Dundee Can be used to produce higher-level products without need for scientific input/expertise Potential products include – Atmospheric – cloud top pressure & temperature, total precipitable water Terrestrial/Earth – reflectance, soil moisture, surface temperature The resource provides processing capability to use these software packages for the first time to create a wider range of products And allow web server/website upgrade to cope with more products and user demand

11. Why this addresses Big Data issues Again – new satellites have higher data rates and produce more data than DSRS has handled in the past E.g. NPP yields data volumes 20x those of Terra & Aqua Processing times and delay to product availability have increased significantly as a result Existing resource limitations prevent higher level processing The asset will address these issues by Enabling more efficient processing of larger volumes of data Increasing speed from reception to data/image product availability Provide resources to exploit available processing packages to increase variety of products

12. How it fits with Big Data infrastructure DSRS data and products could be made available for use within CEMS/JASMIN and the Satellite Applications Catapult CEMS/JASMIN facilities and tools might be used for processing and analysis of our data, e.g. entire archive could be processed efficiently for long time-series requirements DSRS can become a data provider to the Satellite Applications Catapult, e.g. possible development of applications and services by organisations working with the Catapult Initial discussion between DSRS Director and Prof Nick Veck, Satellite Catapult – Interest in DSRS providing data for potential near-real time use

13. Case for timely procurement No open tender delay – cost below OJEU threshold and purchased via existing Uo Dundee supplier contract Latest quote indicates under 10 days from order to delivery Equipment installation and initial configuration – complete around 6 weeks from funding allocation Application software installation, migration to new web servers, initial product creation/testing ~ 6 weeks further Reach this stage 2-3 months before 31 March 2014 deadline Begin publicising new products to potential users More products added – through and beyond March 2014

14. Response to outline panel feedback Generic points – relevance to Big Data as previously indicated Capability to process larger volumes of data than before Process these much more quickly Increase variety of products we can provide Specific feedback – none received – panel agreed that proposal Aims at a large user group Has strong scientific need

15. Thank you!