1 / 10

Advancements in HDF5/SRB Integration for Enhanced Astrophysical Data Management

This report presents the current status of the HDF5/SRB integration project, highlighting significant enhancements in data transfer performance and the planned development of the SRB-FLASH architecture. Key achievements include reduced transfer times across various data sizes, improvements in GUI components for HDFView, and support for writing HDF5 files in the SRB framework. The project is divided into three phases focused on improved data selection, metadata querying, and developing a versatile framework for future applications. This integration facilitates efficient data sharing and management for the scientific community.

upton
Télécharger la présentation

Advancements in HDF5/SRB Integration for Enhanced Astrophysical Data Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF5/SRB IntegrationAugust 28, 2006 Peter Caoxcao@ncsa.edu HDF, NCSA Mike Wanmwan@sdsc.edu SRB, SDSC Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration

  2. Current Status • Improved transferring raw data (SDSC) • Tested performance test #1 (270MB) from 18 to 15 (seconds) test #2 (550MB) from 231 to 35 (seconds) test #3 (820MB) from 734 to 113 (seconds) • Project plan for SRB-FLASH

  3. Things to move forward • Improve GUI component in HDFView to open remote files (NCSA) • Support primitive writing capabilities for HDF5 files in SRB (NCSA and SDSC) • Write documentation and extensive testing (NCSA and SDSC) • Release the production system and publish documentation (NCSA and SDSC)

  4. SAP/SAC program: FLASH • FLASH is a adaptive-mesh simulation code for astrophysical hydrodynamics • FLASH slice extractor retrieves a specific 2D slice from a 3D AMR (adaptive mesh refinement) Slice (gas temperature) and isosurface (gas density) in particle/mesh simulation

  5. SRB-FLASH: architecture

  6. SRB-FLASH: benefits • No need for large storage on local machine. Terabytes of data files sit only on the sever machine • Instant access to the data and metadata. Since only the selected data/metadata are transferred to the client instead of the whole multi-terabyte file, time on the network is tremendously reduced • Data sharing among scientists. Since data sits on the server machine, it can be shared with scientists without multiple local copies of the same file. Also, everyone will have access to the updated data after a new simulation run

  7. SRB-FLASH: project plan • This project will be divided into three phases: phase I, phase II and phase III • Phase I and part of phase II are required • Phase III and part of phase II are optional, depending on the duration and amount of the funding • The time estimates are based on an half (50%) FTE

  8. SRB-FLASH: milestone Phase I: a command line client utility and server function to support selection of FLASH AMR mesh data.

  9. SRB-FLASH: milestone Phase II: enhanced utilities querying FLASH specific metadata and statistics, and performing data interpolation and simple mathematic manipulation.

  10. SRB-FLASH: milestone Phase III: development of general framework that can be adapted to other applications. We need to scope out the milestone for this phase based on the experience of phase I and II.

More Related