1 / 13

NERC DataGrid: Bridging Data Silos for Interdisciplinary Research

Explore the journey of NERC Data Grid from its inception, modular design, metadata management, to future goals like ESG/VCDAT integration. Discover how GridFTP enhances large file transfers for NERC data collaboration.

kimo
Télécharger la présentation

NERC DataGrid: Bridging Data Silos for Interdisciplinary Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The NERC DataGrid: Making data interdisciplinary! Bryan Lawrence (BADC) Roy Lowry (BODC), Kerstin Kleese van Dam, Kevin O’Neill, Andrew Woolf (CLRC), Dean Williams (PCMDI) and the rest of the NDG team.

  2. Outline • Brief intro to the NDG goals: • Fat clients, and inter-disciplinarity … • Evolution: from BODC&BADC to the wider community. • Modular Design • Keeping track of the metadata • Harvesting • Where we are now …(we only started in Sep02) • What lies ahead?

  3. ESG/VCDAT: Example of a Client Application • We will: • Provide python based classes for our observational data to complement the access to 3D gridded data. • It will be possible to overlay model and observational data using grid tools.

  4. The NERC Data Grid & the NERC Metadata Gateway

  5. The Data Use Chain

  6. NDG: Required “Data” Metadata Need a tool to generate B!

  7. NDG Architecture, status June 2003

  8. Using Globus GridFTP Statement: GridFTP is a “faster-stronger” FTP. Is it? Yes: tests between DL and RAL and RAL and POL suggest an average factor of two in performance for large file transfers (although peak FTP rates can reach ¾ of GridFTP). And: network reliability and bandwidth at 2 Mbit/s is not good enough for sustained large file transfer without GridFTP! (500 Mbyte file requires 40 minutes cf 80 minutes). But: 2 Mbit/s is too slow to deal with the file sizes of interest! Big problem for NERC … The same 40 minute file would take under a minute between DL and RAL • Means a different trade off between client and server processing • Fat clients for Fat pipes, Thin clients for Thin pipes.

  9. Under-the-hood, the power of XML … ECMWF ERA40 • Many TB in spectral format • Double that in NetCDF! • Want to avoid using tape-drives! We have: • Implemented a new caching system based on CDAT to do “on-demand” conversion from spectral-NetCDF • Information in LAS database drives the CDAT back-end, if possible use existing NetCDF otherwise convert on-the-fly. Next: • Will link this and other datasets to our intermediate schema. • Need to add Met Office data (working on new drivers).

  10. Problems ahead? • Access control: technical issues, policy, social habits, trust. • Quality of existing metadata, how to collect what we need? • Joining it all together, Joining all us together … • Making sure we are OGC, ISO compliant. • Making it robust … • (Ten times as much effort as making it work, Tony Hey, June 2003)

More Related