1 / 8

Automated Data Pipeline for Improved Output Quality

Enhance data quality with automated data classification, calibration association, and robustness. Streamline data packages generation for optimal results. Collaborate interactively to ensure calibration accuracy and user satisfaction.

mirit
Télécharger la présentation

Automated Data Pipeline for Improved Output Quality

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. General Introduction Yves Jung

  2. Pipeline Usage • Automatically in Paranal on a dedicated platform (Instrument Scientist) • Automatic data classification • Automatic association with the calibration data • Requires robustness • Interactively in Garching to deliver properly calibrated data packages (DFO group contact person) • Better calibration data association improves data products quality • Most extensive usage brings most of the feedback • External user once the public release has been achieved • Only possible when the pipeline has been validated • Usually from Gasgano • Recipes need to be clean (no memory leak allowed)

  3. Pipeline Requirements • A recipe should never crash ! • Proper errors handling • Meaningful error message required • No memory leaks allowed ! • Easy to check (use --mem-check esorex option) • Clean code makes it easy to avoid them • A recipe should run faster than the input data exposure time

  4. Pipeline Packages • Dictionaries in iiinstrument-dic-x_y_z.tar.gz • Defines QC keywords: ESO-DFS-DIC.IIINSTRUMENT_QC • Defines raw files header keywords: ESO-VLT-DIC.IIINSTRUMENT_ICS, ESO-VLT-DIC.IIINSTRUMENT_OS, etc… • Configuration and Calibration files in iiinstrument-calibdb-x_y_z.tar.gz • Calibration data for the automatic data reduction in Paranal • Association rules between data and reduction recipes • Configuration files for Gasgano • Data Reduction Package in iiinstrument-x_y_z.tar.gz • Instrument specific data reduction library • Recipes • Pipeline user’s manual (for the public release)

  5. Pipeline Dependencies • FITS IO • Currently QFITS 6.2.0 (http://www.eso.org/projects/aot/qfits) • CFITSIO replaces QFITS from CPL 4.0 on • Common Pipeline Library • Current version 3.1 (http://www.eso.org/cpl) • Version 4.0 planned for October 2007 • Recipes execution • Esorex (currently 3.6.1 from http://www.eso.org/observing/cpl/esorex.html) • Gasgano (currently 2.2.4 from http://www.eso.org/observing/gasgano) • Additional dependencies (See C. Izzo presentation)

  6. Pipeline Template • iiinstrumentp.tar.gz is a pipeline template • Instrument name: IIINSTRUMENT • Recipe name: rrrecipe • Version number 0.0.1 • Implements the standard way of reading keywords, loading and saving data (See S. Castro presentation), … • Can be installed and executed without modification • Should be the starting point for a pipeline project • The Technical Developers manual describes the different parts of the template • The template contains two main subdirectories • iiinstrument/ contains the instrument specific data reduction library • recipes/ contains the recipes (that usually call functions implemented in the library)

  7. First steps • Read the Deliverables Specification for VLT/VLTI Instruments document (VLT-SPE-ESO-19000-1618) • Read the Technical Developers manual (VLT-MAN-ESO-19500-3349) • Install QFITS, CPL, ESOREX, GASGANO, autotools (See R. Palsa pres.) • Unzip/untar iiinstrumentp.tar.gz • Rename iiinstrument, Iiinstrument, IIINSTRUMENT, rrrecipe everywhere (Use a 5 letters name for iiinstrument !!!) • Put the new package under CVS • Follow the development plan with your contact at ESO/SDD/DFS

  8. During the development • Pay attention to the memory handling (See J. M. Larsen presentation) • Follow unit tests procedures (See L. K. Lundin presentation) • Run acceptance tests (See A. Modigliani presentation)

More Related