Ada95 Bindings for the NCSA Hierarchical Data Format (HDF) Libraries

1. Ada95 Bindings for the NCSA Hierarchical Data Format (HDF) Libraries Bruce R. Barkstrom Atmospheric Sciences Data Center NASA Langley Research Center Hampton, VA

2. HDF Early History • HDF was in the original MOSAIC browser from the National Center for Supercomputing Applications (NCSA) • HDF Goals • Provide uniform and platform-independent file format for large arrays • Provide self-documenting file format for scientific data

3. NASA Interest in HDf • 1990 – NASA preparing to design data system for Earth Observing System • Large number of instruments on many satellites~ 20 instrument teams • Very large amount of archival data~ 1 TB/day now; 3 TB/day next year • HDF selected by EOSDIS project as “standard format”

4. Reasons for NASA Interest • Self-documenting format • Embed documentation with binary values • Multi-platform support • NCSA routinely tests on scientific computing platforms: Crays to PC’s (UNIX and Windows) • Machine independence • Write or read with transforms from little-endian to big-endian or back • Support for data compression • General enough to be common across communities

5. Conventional Ada95 Approach to Files • Four methods supported: • Text I/O – ASCII text with strings and support for page formatting • Sequential I/O – Choose a single data type (integer, float, character, record) and move sequentially • Direct I/O – Choose a single data type and an index for locating a record • Streaming I/O – Sequence of objects, with some possibility of access by pointers

6. HDF5 Approach to Files • Heterogeneous data types allowed • Usual data types (integers, floats, strings) • Compound data types (inhomogeneous records) • Objects in file are typed • Arrays described by special structures in file (Datatype and Dataspace) • Objects are accessible by name • Quasi-random object access on write or read

7. Building Ada Bindings for HDF • HDF Libraries are essentially Open Source • Available from NCSA HDF Group – for C(++) and FORTRAN, both precompiled (for common machines) and source • On-line documentation • On-line help available • Different machines supported by C pre-processor directives • Choose to ask John S. Walker (ObjectAda equivalent to MFC) to translate C++ header files • Testing bindings by building Ada equivalents to HDF Tutorial example programs

8. HDF5 Binding Contents • 6 Ada Files • HDF5_VCTypes: numerical type equiv to C • subtype int is Interfaces.C.int; • HDF5_VC.ads: symbol table types • subtype int is HDF5_VCTypes.int; type Int_ptr is access all Int; type Int_array is array (HDF5_VC.Int range <>) of aliased Int; • HDF5_VC.adb: null package body

9. HDF5 Binding Contents (Cont’d) - HDF5Constants: library constants SIZEOF_DOUBLE : constant := 8; - HDF5C.ads: types and function specs subtype size_t is HDF5_VC.Int; subtype size_t_ptr is HDF5_VC.Int_ptr; subtype size_t_array is HDF5_VC.Int_array; - HDF5C.adb: type initializations HDF5Constants.H5T_INTEL_I32 := HDF5Constants.H5T_STD_I32LE; HDF5Constants.H5T_INTEL_F32 := HDF5Constants.H5T_IEEE_F32LE; HDF5Constants.H5T_INTEL_F64 := HDF5Constants.H5T_IEEE_F64LE;

10. Binding Operation • Use C version of HDF5 libraries • HDF5C uses pragma Importto connect Ada to CH5T_IEEE_F32LE_g : aliased HDF5Constants.hid_t;pragma Import (C, H5T_IEEE_F32LE_g, External_Name=>"H5T_IEEE_F32LE_g"); • Write Ada code with Ada form of library function calls • Link compiled code with HDF compiled libraries to produce executable

11. Example Code with Text_Io; use Text_Io; with HDF5_VC; use HDF5_VC; with HDF5Constants; use HDF5Constants; with HDF5C; use HDF5C; with interfaces.c.strings; procedure H5_Crtfile is use type interfaces.c.strings.chars_ptr; --------------------------------------------------------------------- -- Generic Package Instantiation for output to command line interface --------------------------------------------------------------------- package Lio is new Text_Io.Integer_Io(Integer); ------------------------------------------------------------------- -- Variables ------------------------------------------------------------------- fn : interfaces.c.strings.chars_ptr := Interfaces.c.strings.new_string(str => "file.h5"); flags : HDF5_VC.Unsigned := HDF5Constants.H5F_ACC_TRUNC; create_plist : HDF5Constants.hid_t := HDF5Constants.H5P_DEFAULT; access_plist : HDF5Constants.hid_t := HDF5Constants.H5P_DEFAULT;

12. Example Code (Cont’d) File_Id : HDF5Constants.Hid_T; -- HDF5 File Handle -- Status : HDF5C.Herr_T; begin -- Create a new file using default properties. File_Id := H5Fcreate(filename => fn, flags => flags, create_plist => create_plist, access_plist => access_plist); -- Terminate access to the file. Status := H5fclose(File_Id); -- Put status of operations on command line screen put_line("Status of H5Fclose"); if integer(status) >= 0 then put("Status was >= 0 : OK"); else put("Status was <0 : Bad"); end if; lio.put(integer(Status)); put_line(""); end H5_Crtfile;

13. Cares and Cautions • Approach to HDF Bindings requires familiarity with C • Uses access variables and access functions • No exploration of HDF Libraries for machines outside of Wintel (although HDF Group supplies some tools in Java) • Header translation skips library macros and preprocessor definitions (ifdef …)

14. HDF5 Object Structure • Arrays kept in Datasets • Array requires two definitions: • Dataspace: array rank, current dimension, maximum dimension • Datatype: simple (e.g. H5T_INTEL_I32) or compound • Datasets kept in Groups • Groups can be members of other groups • Groups and Datasets can have Attributes

15. HDF5 Structures Groups arranged somewhat like UNIX directory – but hard and soft linking creates graph, not tree: HDF5 "groups.h5" { GROUP "/" { GROUP "MyGroup" { GROUP "Group_A" { DATASET "dset2" { DATATYPE { H5T_STD_I32LE } DATASPACE { SIMPLE ( 2, 10 ) / ( 2, 10 ) } DATA { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 } } } GROUP "Group_B" { } DATASET "dset1" { DATATYPE { H5T_STD_I32LE } DATASPACE { SIMPLE ( 3, 3 ) / ( 3, 3 ) } DATA { 1, 2, 3, 1, 2, 3, 1, 2, 3 } } } } }

16. Interesting Operations • Files and objects can be closed after creation and then reopened for additions or modifications • Arrays can be opened and appended to after creation • Large arrays can be created in chunks • Objects and pieces of objects (hyperslabs and array segments) can be extracted without reading the entire file • Group and dataset objects can be navigated without opening arrays (e.g. to produce a table of contents for file)

17. Interesting Features • HDF5 has XML DTD – readily adaptable to automatic parsing and related actions • General readers for HDF5 (and HDF4) will need to deal with heterogeneous “record” structures that are unknown before the file opens.

18. Interesting Extensions • Self-extracting files • Files contain enough structural information to reconstruct “on the fly” • Executables can be regarded as arrays of bytes – and therefore are extractable directly • Objectbase for scientific data • Scientific data are usually “write once – read one user at a time” • No need for record locking - reduces CPU load and facilitates I/O at rates ~raw I/O • Would need addition of SQL-like syntax for queries

19. Publication Plans • Currently working on NASA Publication of Tutorial for Ada Bindings • Expecting to release Ada Bindings near end of year as “Open Source” • Will try to make available in several venues • NASA Publication (CD plus some hardcopies) • HDF Group Contribution • Ada web site