1 / 8

Enhancing HDF5 and SRB Integration for Terascale Data Management

This document outlines the current status and ongoing efforts of the HDF5 and SRB integration project, which is vital for managing terascale data, especially in the context of earthquake simulations and astrophysical research. Key actions include the publication of HDF5 AIP documents, development of the h5ingest command line tool, creation of HDF5 METS template files, and validation processes. The project supports collaborations like the SCEC Terascale Earthquake Simulations and the ENZO project for cosmological hydrodynamics, ensuring efficient high I/O access and storage.

deion
Télécharger la présentation

Enhancing HDF5 and SRB Integration for Terascale Data Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF5/SRB IntegrationJune 14, 2006 Peter Caoxcao@ncsa.edu HDF, NCSA Mike Wanmwan@sdsc.edu SRB, SDSC Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration

  2. Current Status • Publish HDF5 AIP documents • White paper: http://hdf.ncsa.uiuc.edu/hdf-aip-html/ • HDF5 METS template: http://hdf.ncsa.uiuc.edu/hdf-aip-html/hdf5_mets_template.xml • Finish h5ingest command line tool • Create HDF5 METS template file • Validate HDF5 METS document • Work on test suite and bug fix (Peter Cao) • Work on performance improvement (Mike Wan) • Setup a demo server to support SCEC files

  3. Potential SAC Projects • SDSC ENZO project • Enzo, 3D cosmological hydrodynamics code, simulating the process of massive star formation and destruction • HDF5 is used as file format and parallel file I/O access • FLASH Program • The UC/DOE collaboration on creating three-dimensional, virtual reality projections of the cosmic explosions • HDF5 is used for storing the data and high I/O access • SCEC Terascale Earthquake Simulations • Over 100 TB data/year • Collections at SRB – 2.6 million files, 114 Terabytes

  4. TeraShake Surface Seismograms • 4D Array (1.2 TB) • Time (22,728) • Horizontal (3,000) • Vertical (1,500) • Vector Component (3) • Each file: • 22,728 x 3,000 x 5 x 1 • 1,363,680,000 Bytes • TeraShake scenario • 900 files

  5. Example HDF5 File HDF5 File 32-bit float 22,728 3,000 25

  6. File on SRB server

  7. Select a Subset

  8. HDFView

More Related