1 / 23

Enhancing HDF4 Data Access via OPeNDAP: Progress and Challenges

This report outlines the progress made during the HDF Developers' Meeting regarding the use of OPeNDAP to improve access to HDF4 data. It discusses the implementation of new solutions for better data visualization, downloading efficiency, and compliance with standards like CF conventions. Key challenges highlighted include issues with metadata handling and data structure expectations. The report emphasizes the importance of supporting tools like libnc-dap and visualization software. Future work will focus on expanding test suites and improving functionalities for users.

tyrone
Télécharger la présentation

Enhancing HDF4 Data Access via OPeNDAP: Progress and Challenges

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. HDF4 OPeNDAP ProjectProgress Report MuQun Yang and Hyo-Kyung Lee HDF Developers' Meeting

  2. OPeNDAP Review • A software framework that allows • simple access to remote data • Data Access Protocol • Client-server model • Widely used by Earth Science • Community HDF Developers' Meeting

  3. OPeNDAP Reivew Users Hyrax IDV libnc-dap AIRS DAP Visualization Tools OPeNDAP Clients OPeNDAP Servers HDF4 Files

  4. OPeNDAP Concept Typical Remote Access View Data (User) Remote Data (HDF4) FTP/HTTP Visualization Tools (gradsdap) Visualization tools (e.g. grads) Handler (hdf4_handler) Client Library (libnc-dap) Server (apache) DAP Protocol (via http) HDF Developers' Meeting

  5. Goals Better Access for HDF-EOS2 via DAP • Faster download through selection • Quicker analysis through visualization HDF Developers' Meeting

  6. HDTV Problems You bought a HDTV but... • No HD signal • Needs more Channels in HD • Cable Service Provider Problem HDF Developers' Meeting

  7. HDF4 Problems You downloaded a viewer but... • No CF convention • Needs more HDF4 data in CF • Data Service Provider Problem HDF Developers' Meeting

  8. Challenges: HDF-EOS2 HD Signal Challenge • CF convention (e.g. NetCDF model) • Syntax • Clients expect a certain data structure • Semantics • Clients expect some key attributes HDF Developers' Meeting

  9. Challenges: HDF-EOS2 More Channels in HD Challenge • DVD/VHS/VCD/LD >>> Blue-Ray • Can all titles be delivered in HD? • HDF4/HDF-EOS2 >>> HDF5/HDF-EOS5 • Can all HDF-EOS2 be delivered in CF? HDF Developers' Meeting

  10. Solution: New HDF4 Handler HD Signal Solution • Syntax • Generate the correct DAP data type. • Add shared geo-location information. • Semantics • Insert additional key attributes. HDF Developers' Meeting

  11. Solution: New HDF4 Handler More Channels in HD Solution • Use Metadata parser • Pro: simple • Con: limited channels • Use HDF-EOS2 library • Pro: maximum channels • Con: install HE2 library first HDF Developers' Meeting

  12. Solution Comparison HDF Developers' Meeting

  13. Quick Implementation • Parse Metadata – hdf5_handler • Use HDF-EOS2 library – h4toh5 IDL ncBrowse Ferret MATLAB NCL IDV GrADS ODC HDF Developers' Meeting

  14. Current Status • MODIS / AIRS Grid and Swath • Other Projections (Polar, Sinusoidal) • Testing IDL ncBrowse Ferret MATLAB NCL IDV GrADS ODC HDF Developers' Meeting

  15. HDF-EOS2 in CF via DAP • Finally, Happy Data Consumer! AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdf http://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10 AIRS Swath ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G08331121849.hdf MODIS Grid MOD08_D3.A2008284.005.2008289105354.hdf http://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html MODIS Swath MOD06_L2.A2001222.0025.004.2003127111841.hdf http://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.html HDF Developers' Meeting

  16. Limitations – Missing Attributes • MODIS Grid – Rain fall • AURA OMI Grid - Ozone No Unit! HDF Developers' Meeting

  17. Limitations – Missing Names • How to Match Dimension Names? HDF4 Handler pressureStd → Level NetCDF4 Handler HDF Developers' Meeting

  18. Limitations – Missing Values • How to Supply Missing Dimension Values? MISR_AM1_TC_CLASSIFIERS_P020_O000995_F06_0011.hdf HTTP/1.0 200 OK XDODS-Server: 3.7.9 XOPeNDAP-Server: 3.7.9 XDAP: 3.1 Content-Type: text/plain Content-Description: dods_error Cache-Control: no-cache Error { code = 1002; message = "An internal error was encountered: This HDF-EOS2 file has multiple Grids with different dimension sizes. Please report this to support@opendap.org "; }; HDF Developers' Meeting

  19. Solution – Customization • Provide an option for handler customization to generate fully CF-compliant output • Create a separate XML input file for handler and feed it into the handler. • Insert attributes, renaming rules, and missing values by putting them in the XML input. • This requires a knowledge about the characteristics of data in advance. HDF Developers' Meeting

  20. Solution – Customization • Example: Attributes, Renaming, Values <xml> <variable name=‘RrLandRain’> <attr name=‘unit’ type=‘String’> inches</attr> <attr name=‘_FillValue’ type=‘Int16’>-1</attr> </variable> <rename from=‘GeoXTrack’ to=‘lon’ ></rename> <rename from=‘pressStd’ to=‘lev’></rename> <insert name=‘NAltitudeDim’ size=10 type=‘Int8’> 0,1,2,3,4,5,6,7,8,9 </insert> </xml> HDF Developers' Meeting

  21. Summary • HDF4 Access via OPeNDAP • Is Easy, Efficient and Fast • Is Cool like HDTV • Adding CF is not straightforward • Use HDF-EOS2 library • Is Essential • Maximize the data coverage HDF Developers' Meeting

  22. Future Work (Short Term) • Expand test-suite with data pool • Release through OPeNDAP website • Support for Swath via Parsing • Support for Point data type via HDF-EOS2 library HDF Developers' Meeting

  23. Future Work (Long Term) • General HDF4 Files Support • Customization for CF option • HE2 to DAP Mapping Documentation HDF Developers' Meeting

More Related