1 / 60

HDF Update

HDF Update. Mike Folk The HDF Group The 13th HDF and HDF-EOS Workshop November 3-5, 2009. Topics. What’s up with The HDF Group?. What is The HDF Group And why does it exist?. The HDF Group. Established in 1988

bmartinez
Télécharger la présentation

HDF Update

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF Update Mike Folk The HDF Group The 13th HDF and HDF-EOS Workshop November 3-5, 2009 HDF/HDF-EOS Workshop XIII

  2. Topics HDF/HDF-EOS Workshop XIII

  3. What’s up with The HDF Group? HDF/HDF-EOS Workshop XIII

  4. What is The HDF Group And why does it exist? HDF/HDF-EOS Workshop XIII

  5. The HDF Group • Established in 1988 • 18 years at University of Illinois National Center for Supercomputing Applications • 4 years an independent non-profit company “The HDF Group” • The HDF Group owns HDF4 and HDF5 • Basic HDF4 and HDF5 formats, libraries and tools are open and free HDF/HDF-EOS Workshop XIII

  6. Data challenges addressed by HDF Our ability to organize complex collections of data Efficient and scalable data storage and access A growing need to integrate a wide variety of types of data Long term preservation of data HDF/HDF-EOS Workshop XIII

  7. The HDF Group Mission To ensure long-term accessibility of HDF data through sustainable development and support of HDF technologies. HDF/HDF-EOS Workshop XIII

  8. Goals Maintain and evolve HDF for sponsors and communities that depend on it Provide support to the HDF communities through consulting, training, tuning, development, research Sustain The HDF Group for the long term to assure data access over time HDF/HDF-EOS Workshop XIII

  9. The HDF Group Services • Helpdesk and Mailing Lists • Available to all users as a first level of support • Standard Support • Rapid issue resolution and advice • Consulting • Needs assessment, troubleshooting, design reviews, etc. • Training • Tutorials and hands-on practical experience • Enterprise Support • Supporting many HDF activities across organizations • Special Projects • Adapting customer applications to HDF • New features and tools • Research and Development HDF/HDF-EOS Workshop XIII

  10. Members of the HDF support community • NASA – EOS • NOAA/NASA/Riverside Tech – NPOESS • Army Geospatial Center • A leading U.S. aerospace company • NIH/Geospiza (bio software company ) • University of Illinois/NCSA • Sandia National Laboratory (2) • Lawrence Berkeley National Lab • Projects for petroleum industry, vehicle testing, weapons research, others • “In kind” support HDF/HDF-EOS Workshop XIII

  11. Some areas of increased recent interest • Improvements • Concurrent access • Parallel I/O performance • Real-time write performance • High level language support • Life sciences • Sequencing • Biomedical imaging • Database integration • Microsoft products (HPC, .NET, others) HDF/HDF-EOS Workshop XIII

  12. Cool recent application-Imageworks’ Field3D The Polar Express Spiderman 3 HDF/HDF-EOS Workshop XIII

  13. Topics HDF/HDF-EOS Workshop XIII

  14. Basic Library Releases HDF5 HDF4 HDF4 HDF/HDF-EOS Workshop XIII

  15. Time-line of the HDF libraries releases HDF/HDF-EOS Workshop XIII

  16. HDF5 1.8.3 minor release (May 09) • New functions • Improve flexibility when traversing external links • Validate object identifier • Enabled data chunk cache properties to be set per dataset (per file in previous releases) • Forward/backward compatibility issues • Modified library to be able to open files with corrupt root group symbol table messages • Also corrects corruption errors if found. HDF/HDF-EOS Workshop XIII

  17. HDF5 1.8.4 minor release (Nov 09) • Modified configure and make process to properly preserve user's CFLAGS and similar environment variables. • Corrected a problem where library would re-write the superblock in a file opened for R/W access, even when no changes were made to the file. HDF/HDF-EOS Workshop XIII

  18. HDF5 1.6 minor releases • 1.6.9 May 09 • Minor bug fixes • Same tools improvements as in 1.8.3 • 1.6.10 Nov 09 • Minor bug fixes • Ability to embed library information in executable binaries • This is a last release of 1.6 series • announced in May 2009 – no response • This is your last chance! HDF/HDF-EOS Workshop XIII

  19. HDF 4r2.4 minor release (Feb 09) • Minor bug fixing, enhancements • New routines to get size of compressed data • Support for C shared libraries • Support for 32-bit version on Mac Intel • Updated docs in HTML and PDF HDF/HDF-EOS Workshop XIII

  20. HDF 4r2.5 minor release (Feb 10) • Minor bug fixes, enhancements • Support for 64-bit version on Mac Intel • Restructured and cleaned up source code for easier maintenance • Changes in versioning • Improves ability to maintain • Becomes similar HDF5 versioning works • Will use major, minor, release and sub-release suffix in the names of the source tar balls • E.g., hdf-4.2.5, hdf-4.2.5-snap0 • Library string will include suffix • E.g., "HDF Version 4.2 Release 4-snap3, October 18, 2009" HDF/HDF-EOS Workshop XIII

  21. H4-H5 Conversion Software 2.1 (Feb 09) • Based on HDF4r2.4 and HDF5-1.8.2 • h4toh5 utility • Recognizes HDF-EOS2 files (--with-hdfeos2 configuration option) • Can generate HDF5 files that can be read by netCDF-4 • h4toh5 library • Bug fixes • Performance improvements • http://hdfgroup.org/h4toh5/ HDF/HDF-EOS Workshop XIII

  22. H4-H5 Conversion Software 2.2 (Feb 10) • Based on HDF4r2.5 and HDF5-1.8.4 HDF/HDF-EOS Workshop XIII

  23. Topics HDF/HDF-EOS Workshop XIII

  24. Major Improvements for Existing Tools • H5dump additions • Ability to show data pointed to by dataset region references. • More options for dumping data into ASCII • Compatible with MS Excel • Compatible with h5import • h5diff • Improvements in accuracy, flexibility, and performance • Some new flags • Report non-comparable objects • Avoid NaN detection • Option to use system epsilon to compare floating-point numbers • Compares for strict equality first to improve performance • Treats two INFINITY values as equal • Fixed segmentation fault problem on variable length strings. HDF/HDF-EOS Workshop XIII

  25. Major Improvements for Existing Tools • h5stat • Fixed incorrect statistics on EOS big data files with corrupted headers. • h5repack • Added ability to preserve group creation order • When chunk size not specified, uses heuristics to set chunk size • Fixed problem that 1.8 fails on a file created with 1.6. HDF/HDF-EOS Workshop XIII

  26. Tool activities in the works • New tool -- h5tail • Display new records appended to a dataset • Improved code quality and testing • Tools library: general purpose APIs for tools • Tools library currently only for our developers • Want to make it public so that people can use it in their products HDF/HDF-EOS Workshop XIII

  27. Conversion Tools Please send us your comments and requests regarding HDF5 conversion tools, such as • HDF4 to HDF5 • HDF5 to jpeg • HDF5 to XML • HDF5 to other formats? HDF/HDF-EOS Workshop XIII

  28. Topics HDF/HDF-EOS Workshop XIII

  29. HDF-Java 2.6 is on the way • Includes all HDF java products • Java Wrapper API • Java Object API • HDFView • Adds new features, such as better support for dataset region references • Improves performance • Release schedule • Beta 1: end of Nov. 09 • Full release: end of Dec. 09 HDF/HDF-EOS Workshop XIII

  30. Full support of HDF5 1.8.x in hdf-java • Full HDF5 1.8 support will be added to the release after version 2.6. • We are looking for input • RFC: http://www.hdfgroup.uiuc.edu/RFC/HDF5/hdf-java/ • Java wrapper will be completed March 2010 • Object API and HDFView update to come later HDF/HDF-EOS Workshop XIII

  31. Topics HDF/HDF-EOS Workshop XIII

  32. Single-Writer/Multiple-Reader Access • Situation: A long-running process is modifying an HDF5 file and simultaneously other processes want to inspect data in the file. • Solution: Single-Writer/Multiple-Reader (SWMR) File Access. • Allows simultaneous reading of HDF5 file while the file is being modified by another process • No inter-process coordination necessary HDF/HDF-EOS Workshop XIII

  33. Improved Multi-Threaded Concurrency • Converting from “big lock” on code (entire library) to locks on internal library data structures • Will improve ability to have multiple threads performing HDF5 operations simultaneously HDF/HDF-EOS Workshop XIII

  34. Other Library Features • Saving space • Store Partial Edge Chunks More Efficiently • Persistent File Free Space tracking/recovery • Allow a group’s link info to be compressed • Saving time • Aggregate neighboring metadata for faster metadata cache I/O HDF/HDF-EOS Workshop XIII

  35. New chunk indexing methods HDF/HDF-EOS Workshop XIII

  36. Project with Lawrence Berkeley Nat’l Lab to improve HDF5 performance on parallel applications Up to 6x performance improvements on certain applications (so far) Parallel I/O Improvements HDF/HDF-EOS Workshop XIII

  37. Topics HDF/HDF-EOS Workshop XIII

  38. HDF-EOS library HDF/HDF-EOS Workshop XIII

  39. EOS support • HDF-EOS2 and HDF-EOS5 • Automatic configuration with szip enabled/disabled • Now tested daily with HDF4 and HDF5 development code • Updated the HDF-EOS website HDF/HDF-EOS Workshop XIII

  40. HDF-EOS5/netCDF-4 Augmentation Tool Accessing HDF-EOS5 files via netCDF-4 API HDF/HDF-EOS Workshop XIII

  41. The Main Challenge GRIDS HDFEOS CloudFractionAndPressure Data Fields • Would like netCDF-4 applications to be able to read and understand HDF-EOS 5 files • Problem: NetCDF-4 model follows the HDF5 dimension scale model but HDF-EOS5 does not. CloudFraction CloudPressure No HDF5 dimension scales are associated with this variable HDF/HDF-EOS Workshop XIII

  42. Our Solution – Augmentation GRIDS HDFEOS CloudFractionAndPressure Data Fields • Provide dimensions required by netCDF-4 XDim YDim CloudFraction[XDim][YDim] CloudPressure[XDim][YDim] HDF/HDF-EOS Workshop XIII

  43. Special values in HDF5 • There are cases where a user may wish to specify more than one “special” value to describe non-standard data. • We provide several examples (C, Fortran, IDL) on how to store special values • http://www.hdfgroup.org/pubs/rfcs/ HDF/HDF-EOS Workshop XIII

  44. OPeNDAP HDF/HDF-EOS Workshop XIII

  45. OPeNDAP • HDF5-OPeNDAP handler • Served OMI Swath data • HDF4-OPeNDAP handler • Tested with some AIRS data and some MODIS data • More information in the Thursday morning session HDF/HDF-EOS Workshop XIII

  46. Swath to Grid conversion Tool • Request from NASA GES DISC • Convert Swath to Grid • Support both HDF-EOS2 and TRMM data • Still in the development MODIS Swath Converted Grid HDF/HDF-EOS Workshop XIII

  47. Support for NPP/NPOESS by The HDF Group HDF/HDF-EOS Workshop XIII

  48. Priorities for 2008-2009 • Data accessibility and usability • Developed library of high level APIs to support NPP/NPOESS data management • Modified h5dump to display region references • Modified HDFView to view object and region references and quality flags • System maintenance • User support HDF/HDF-EOS Workshop XIII

  49. NPOESS Project Information • Project Web site • http://www.hdfgroup.org/projects/npoess/ HDF/HDF-EOS Workshop XIII

  50. HDF4 Layout Maps HDF/HDF-EOS Workshop XIII

More Related