AeroADL

1. AeroADL Scott Houchin Project Leader, VIIRS SDR analysis and data access tools scott.houchin@aero.org, 571-307-3914 Engineering & Technology Group, Electronics and Sensors Division 15 November 2012

2. ADL from an analyst viewpoint • ADL is a powerful developer toolkit for running the operational algorithms on a desktop system Developer loop Gather inputs Run algorithm Examine outputs Modify code Analyst loop • For analysts, every run is different • Different RDR or other science data input • Different LUT or processing coefficients • Staging inputs is complicated and error prone

3. Why not “just populate the DMS”? • Analyst still needs to create the ADL execution XML document • Unpacking everything into your own local data management system (e.g. the ADL/data/HDF_Unpack_Area) won’t always work • No quick link back from granule ID to URID • Analyst need to control LUT/input selection beyond granule collection date and effectivity • e.g. Use an experimental VIIRS-SDR-F-LUT

4. Need to simplify use in analyst mode • The goal — Stage and run a set of data with one commandthat can be entered interactively • Automatically create input data directory with either HDF or raw binary inputs • Generate the ADL execution XML document • Control storage of ADL log files such that if something goes wrong, analyst can quickly find the correct log file to examine • Run the actual ADL algorithm or algorithm controller, selecting only the desired portions of the controller Drastically improve process for staging tailored inputs

5. So how did we do? View this slide in slide show mode • Given 3 VIIRS science RDR files:/rdr/RNSCA-RVIRS_npp_d20111128_t0829253_e0830507_b00439_c20111128102607267562_noaa_ops.h5/rdr/RNSCA-RVIRS_npp_d20111128_t0830507_e0832160_b00439_c20111128102729213164_noaa_ops.h5/rdr/RNSCA-RVIRS_npp_d20111128_t0832160_e0833414_b00439_c20111128102901627326_noaa_ops.h5 • Run the ADL VIIRS SDR Controller, starting with RDR data, stopping after calibration, but using an experimental version of the VIIRS-SDR-F-LUT/data/sn-002/VIIRS-SDR-F-LUT_sn002.bin • adl v/control -glrdrtocal -od /data/mytestrun \rdr=/rdr/RNSCA-RVIRS_npp_d20111128_t0830507_…h5 \sdrf=/data/sn-002/VIIRS-SDR-F-LUT_sn002.bin adl command VIIRS SDR Controller algorithm RDR to Calibration GuideList Staging directory Nominal Science RDR file adl v/control -glrdrtocal -od /data/mytestrun rdr=/rdr/RNSCA-RVIRS_npp_d20111128_t0830507_…h5 sdrf=/data/sn-002/VIIRS-SDR-F-LUT_sn002.bin F LUT override

6. Behind the scenes, one time setup is required • Additional GuideLists predefined in $AEROADL_HOME/cfg for different algorithm selections • Algorithm input templates predefined for each GuideList • Database of available versions of required LUT/PC files (e.g. VIIRS-SDR-GAIN-LUT) • Indicators, by collection short name of dynamic inputs • Required inputs (e.g. VIIRS-SCIENCE-RDR) • Optional inputs, (e.g., VIIRS-SOLAR-DIFF-AGG-HISTORY-AUX) • Modification to Log output directory configuration options • Modification to algorithm configuration to persist intermediate products to disk • Tools to pull granule metadata out of RDR files • Several other helper tools

7. So what happens right when you run adl • From your abbreviated command and GuideList tokens, adl determines the actual executable and GuideList • Locates input template for command/GuideList combination and catalogs template contents • Wraps binary input overrides with an ADL metadata file and URID • Creates a staging directory, into which is put • Links to any default input files or auxiliary input directories that were not overridden • The unpacked contents of input HDF files • Symbolic links to your original binary input files, along with an ADL metadata file describing your binary input • Generates the algorithm execution XML document • Loads the ADL environment and runs the correct ADL executable

8. Staging directory contents • config.xml— The algorithm execution XML document • GuideList.cfg— A link to the specific GuideList document used, for recordkeeping purposes • inputs/ — All of the inputs specific to this ADL run for traceability • Links to files from the template • Actual files for unpacked inputs • outputs/ — The directory into which ADL will write your outputs • logs/ — The directory into which ADL will write all logs related to this ADL instance • run.sh— A script allowing you to rerun this exact instance of ADL inputs, either for recordkeeping or for development purposes

9. Sensor/algorithm support • Fully tested on VIIRS SDR from RDR through calibration • GuideLists: ProSdrViirsVRdr, ProSdrViirsRdrToGeo, ProSdrViirsRdrToCal, ProSdrViirsCal, Default • Fully tested on VIIRS Solar Diffuser Module • User must provide 10 granules with sun out of view and with dates greater than granules with sun in view • Support for other sensors and algorithms anticipated in the code • Partial OMPS support confirmed • Updated needed to deal with TC vs NP in algorithm names • Other sensors likely better supported • Input templates for other algorithms needed • Potential issues when more than just simple inputs and granule IDs • ADL 4.0 and greater • Only withMetadata mode supported

10. LUT and processing coefficient updates • LUT updates • Custom templates for AeroADL organized to simplify keeping track of of current versions of LUTs • Will be updated as new versions of LUTs move into production • PolarWander • Service running to get updated PolarWander data and convert to the ADL binary files, integrated into AeroADL templates • Modified version of Paul Meade’s PWConverter • Produces big endian files

11. System configuration • Official ADL environment variables not loaded upon login • Prevents any possible conflicts with other tools (e.g. $GCCLIB, $PATH, $JAVA_HOME, …) • envSetup loaded at time of execution by AeroADL scripts • Both Bash and CshenvSetup variations supported • Installation specific cshrc and profile files provided • Source either file depending on your shell • Configuration files automatically determine where installation is located • Several other environment variables specific to AeroADL must be defined • See “doc/AeroADL Install.doc” and “adl -help” for full documentation

12. Auxiliary scripts • addtodms— “Unpack” a new LUT into the correct portion of the DMS • adl_pack, adl_unpack— Run the ADL Packer or Unpacker, controlling log file location • lut_unpack— Wrap a binary LUT file with an ADL metadata file and a URID • csn— Extract the collection short name from the name of a file • genurid— Generate a URID, optionally excluding the time from the output • linkbycsn— Input template creation helper to link a file by the CSN extracted from the filename • mkoptinput, lsoptinputs— Helpers for adding optional input indicators into a template • makeadl— Load ADL environment before running make

13. Examples • Produce a Verified RDR file • adl v/control -glvrdr -odvrdroutrdr=RNSCA-...h5 • Run 3 consecutive RDR files through Geo • Specify the nominal RDR and AeroADL will find the +/-1 files • adl v/control -glrdrtogeo -odgeooutrdr=RNSCA-...h5 • Run your previously generated Geo output through cal • adl v/control -glv/cal -odcalout -cipgeoout • Run 3 consecutive RDR files through Cal • adl v/control -glrdrtocal -odcaloutrdr=RNSCA-...h5 • Run a directory of OBC-IPs (already unpacked into .bin and .asc files) through the solar diffuser • adl v/solar -odsolarout -cid obcipdir -gidsobcipdir/*OBC-IP* -co

14. AeroADL available on GRAVITE • At /apps/user/aero/aeroadl/latest • Source the desired cshrc file from in your ~/.cshrc to load the environment • source /apps/user/aeroadl/latest/etc/cshrc \/apps/user/aeroadl/latest/etc/cshrc • (tcsh scripts cannot determine where they are located from the script itself when source from your ~/.cshrc, thus you must put the path to the script as a second parameter) • PolarWander database • /apps/user/aero/aeroadl/latest/DMS/USNO-PolarWander-UT1-ANC-Int

15. Installing on your own system • Available through GRAVITE Subversion • https://svn.gravite.gov/aeroadl/trunk (Full VPN) • https://localhost:4443/aeroadl/trunk (WebVPN) • COTS dependencies and sample data not included • You will need to build the COTS dependencies and ADL as part of your install • See “docs/AeroADL Install.doc” for installation instructions • Updated PolarWander files: • https://svn.gravite.gov/aeroadl/trunk/DMS/USNO-PolarWander-UT1-ANC-Int • Update directory using “svn update”

16. Other Aerospace tools • VIIRS Science RDR and SDR readers • Read HDF or unpacked binary files • Java library with wrappers for use in IDL or MATLAB • Download through CasaNOSA subversion or use on GRAVITE • https://casanosa.noaa.gov/svn/viirs_early_release/trunk/vsdrp/vsdrp-rdr-sdr • /apps/user/aero/vsdrp/vsdrp-rdr-sdr/latest • Binary LUT readers/writers • Read and write values into the fixed length binary LUT/PC files • Java library with wrappers for use in IDL or MATLAB • Supports VIIRS SDR, easily extensible to other files • Download through CasaNOSA subversion or use on GRAVITE • https://casanosa.noaa.gov/svn/viirs_early_release/trunk/vsdrp/vsdrp-lut • /apps/user/aero/vsdrp/vsdrp-lut

17. Support • Officially provided by The Aerospace Corporation “as is” • Informal support is available by contacting Scott Houchin • VIIRS SDR and Solar Diffuser is high priority • Others are much lower • Collaboration with others on both code and template updates is welcome