80 likes | 93 Vues
Develop a service that reduces labor effort to create user-specific tailored products from HDF and HDF-EOS files by providing routine data manipulation and configurable output options.
E N D
NDE XML Tailoring Service HDF and HDF-EOS Workshop XIII November 4, 2009 Tom Feroli, Peter MacHarrie
Goal • Reduce the labor effort to create user-specific tailored products (I.e., image, gridded, map reprojected, and other specific file formats, etc.) • Objective • Develop (InputFormat + Output Format) software applications instead of (Input Format * Output Format) software applications. • Approach • Develop a service that provides routine data manipulation (i.e., sub-setting, sub-sampling, filtering, aggregation) and places output into a well-defined intermediate format. • Create tailoring applications that accept a well-defined intermediate format and configurable output options to create a specific end-user product.
Data Selection Driver XML Reads Writes Data Selection Driver Uses Binary Format (Subsample, Subset, Aggregate, Filter, Scale/Offset) HDF5+ netCDF4 netCDF4 netCDF3 • Modular design, accepts drivers to accommodate additional input • and output file formats (e.g., HDF-EOS5, netCDF3, etc.)
Enterprise “Measure” • Enterprise Measure is a logical construct • It is the concatenation of all related arrays in all files in an “enterprise” • File 1 – Array1 (Scan = 12, Sample=96, Channel=22) • File 2 – Array1 (Scan =12, Sample = 96 , Channel=22) • Enterprise Measure1 (Scan=24, Sample=96, Channel=22) • Enterprise Measure reference “Enterprise Dimensions” • A file, is a storage unit, for a partition, of the Enterprise Measure • The partition is a range of values of one or more Enterprise Dimensions • File 1 – Scan Dimension 1 -12 • File 2 – Scan Dimensions 13-24
Enterprise “Dimension” • Enterprise Dimension is a unique logical mapping to dimensions within all data files. • In VIIRS, Image and Moderate product profiles, the dimension AcrossTrack is defined as 6400 and 3200 • Enterprise dimension are named uniquely for each • AcrossTrack-I 6400 • AcrossTrack-M 3200
Enterprise Data Cube • Data Cube is a structure of one or more Enterprise Measures. Within a data cube: • Measures share the same dimensions • Measure names are unique • Measures may have different data types • In VIIRS SDRs all arrays are named Radiance • Enterprise measure uniquely name each array • Example • Radiance_I1 -> Product I1->Radiance Array • Radiance_I2 -> Product I2->Radiance Array
Enterprise Data Structures Tailoring Service XML References Enterprise Data Structures
Measures (additional information) • Named Subset • Array: AntennaTemperature(12, 96, 22) • Measure: AntennaTemperature, references entire AntennaTemperature array • Measure: AntennaTemp-Channel_1, references array AntennaTemperature, with sub-sample specification of Start=(1,1,1), Stride=(1,1,1), Count=(12,96,1) • Named BitMask • Array: QF1_VIIRSCMIP(768, 3200) • Measure: CloudConfidencePixel • Has Integer DataType • References array QF1_VIIRSCMIP starting at bit 2 for 2 bits • Scale and Offset • Measure defined as float, references an integer array and its scale and offset are applied • Measures can be referenced in the <select> or <filter> elements of the tailoring XML. • <where><filter><measure name=‘CloudConfidencePixel’> • <operation value=‘LT’><operand value=‘2’/></filter></where> • (i.e., CloudConfidencePixel < 2)