General Philosophy of Table Driven Data Representation Forms

1. General Philosophy of Table Driven Data Representation Forms Simon Elliott EUMETSAT simon.elliott@eumetsat.int RA-VI Regional Training on BUFR and Migration to Table Driven Code Forms Langen, Germany, 17 - 20 April, 2007

2. Tuesday Late Morning Agenda • Self Description • Code Structures • BUFR and CREX Tables • Features Common to BUFR and CREX • Differences Between BUFR and CREX RA-VI Regional Training on BUFR and Migration to Table Driven Code Forms Langen, Germany, 17 - 20 April, 2007

3. Data exchange and codes • a code: “A system of words arbitrarily used for other words, to secure brevity and secrecy” • a cipher: “A secret manner of writing … intelligible only to those possessing the key” • to cipher: “To express by occult characters” Purpose is global exchange of data in an unambiguous and efficient way. WMO “codes” are data representation forms (DRFs). RA-VI Regional Training on BUFR and Migration to Table Driven Code Forms Langen, Germany, 17 - 20 April, 2007

4. Problems • Increasing volume and complexity of observations • Changing requirements but fixed codes • Change implies long time scale and costs: equipment manufacturers, training et c. • Changes take about 2 years, if essential RA-VI Regional Training on BUFR and Migration to Table Driven Code Forms Langen, Germany, 17 - 20 April, 2007

5. Table driven codes • GRIB - Processed data in the form of grid-point values expressed in binary form. • Forecast, analysis, climatology, satellite images and products • BUFR - Binary universal form for the representation of meteorological data. • Observations, satellite products • CREX - Character form for the representation and exchange of data. • Like simple BUFR but human readable ASCII RA-VI Regional Training on BUFR and Migration to Table Driven Code Forms Langen, Germany, 17 - 20 April, 2007

6. General structure • Identification: GRIB/BUFR/CREX • Header: Date, time, originator, table versions ... • Optional section: Metadata (potentially XML), private data … • Data description: What sort of data follows • Actual data: here • Closure: “7777” Table driven codes generally have this structure  RA-VI Regional Training on BUFR and Migration to Table Driven Code Forms Langen, Germany, 17 - 20 April, 2007

7. Some features • Data are self defining: type, content, edition, section lengths et c. • Header data are at fixed offsets for quick access without decoding • Binary data (GRIB and BUFR) are not really human readable • Optional section can contain anything; use of XML for metadata under continuing discussion RA-VI Regional Training on BUFR and Migration to Table Driven Code Forms Langen, Germany, 17 - 20 April, 2007

8. Consider BUFR • Table B - Element descriptors • Start with ‘0’, temperature, pressure, et c. • Table C - Operator descriptors • Start with ‘2’, change of scale, qc follows, et c. • Table D - Sequence descriptors • Start with ‘3’, wind sequence, high accuracy location sequence, et c. • Code and flag tables - • Type of cloud, channel combination, et c. RA-VI Regional Training on BUFR and Migration to Table Driven Code Forms Langen, Germany, 17 - 20 April, 2007

9. Table B Example: 0-20-082 Amount of segment % 0 0 7 cloud free Element descriptor Entry in class Unit Reference value Class 20, observed phenomena Element name Scale Data width (bits) RA-VI Regional Training on BUFR and Migration to Table Driven Code Forms Langen, Germany, 17 - 20 April, 2007

10. Scale, reference and width Encoding: value = (obs x [10^”scale”]) - “reference” in “width” bits e.g.: Let scale=1, reference=-50, width=7 then, if obs=1.3, encoded value is 63 allows obs from -5.0 to +7.6, steps of 0.1 Decoding: obs = (value + “reference”) ÷ [10^”scale”] e.g.: Let scale=-2, reference=5, width=4 then, if value=14, decoded value is 1900 RA-VI Regional Training on BUFR and Migration to Table Driven Code Forms Langen, Germany, 17 - 20 April, 2007

11. Table C Operator descriptors for more complex tasks, e.g. • Change width • Change scale • Change reference • Quality control follows • First order statistics follow • Define “data present” bit map • Re-use “data present” bit map • Cancel “data present” bit map RA-VI Regional Training on BUFR and Migration to Table Driven Code Forms Langen, Germany, 17 - 20 April, 2007

12. Table D • Not strictly required • Saves space in Section 3 (list of descriptors) … about 1.5% for EUMETSAT winds • One table D entry represents an agreed list of table B and/or C and/or D descriptors • 3-01-011 = 0-04-001 Year 0-04-002 Month 0-04-003 Day RA-VI Regional Training on BUFR and Migration to Table Driven Code Forms Langen, Germany, 17 - 20 April, 2007

13. BUFR is binary, better for telecoms and computers BUFR has a compression scheme BUFR mostly uses SI units (flight level is in meters) CREX is ASCII, easier for humans to read and write CREX has no explicit compression scheme CREX allows other units (temperature can be Celsius or Kelvin) Differences between BUFR and CREX In general, BUFR and CREX are quite similar, the difference being that CREX is easier for human encoders/decoders where computers are not available or telecoms are inadequate ... RA-VI Regional Training on BUFR and Migration to Table Driven Code Forms Langen, Germany, 17 - 20 April, 2007