Improved and automated QC process Reinhard Hanuschik and the DPQC group

1. Improved and automated QC process Reinhard Hanuschik and the DPQC group • fast data transfer: July 2008 • full data stream VLT and VLTI (ex burst-mode data) • > 50% within 10 minutes • new ways in data processing @ DPQC • impact on shared QC • 2 main goals: • calibration completeness • calibration quality

2. Mission #1: calibration completeness (“save last night”) 1. Setting the stage • calChecker • calibrations incomplete: science OBs to be repeated • focus on calibration plan: • known, predictable, configured • can be checked automatically

3. Mission #2: calibration product quality (“save next night”) combining Health Check monitor and scores: quick-look HC monitor

4. any problem not discovered on site: up to 10 days affected strong need for 2 QC processes:daytime astronomer Paranal (preliminary),QC scientist Garching (final) basic real-time QC with opslog data until 2008: data acq (Paranal) data in Garching (after ~ 10 days) opslog

5. now (2009): fast data delivery • fast QC feedback • improved QC feedback • measure quality (QC1 parameters) • compare quality (trending) • rate quality (scoring) new! data acq (Paranal) data in Garching

6. 2. calChecker • www.eso.org/qc/ALL/daily_cal.html • evaluation for 7 last days (columns) and all science setups (rows) • driven by science data • checks: • all science setups for a given instrument • checks calibration plan • knows about validity and required number of files  life examples

7. evaluates each box in colors: • green (OK):all calibrations complete and within validity(e.g.: 5 BIAS & 3 FLATs & 1 ARC & 1 STD within validity) • yellow (NOK):calibrations complete but some outdated(e.g.: as above but 3 FLATs are 2 days old) • red (MISS): calibrations incomplete, at least one missing(e.g.: 5 BIAS & 3 FLATs ok, 1 ARC & 1 STD missing) • missing or outdated calibrations are indicated

8. links to details • for each box: complete list of science files • prog_id, obs_id etc. for easy reference • OB grade and comment: useful for analysis

9. BOCAL, EOCAL • signal about daytime calib queue • calChecker displays “pending”, “ongoing”, “finished” for current date • ANALYSIS • QC scientist to provide analysis/comments, but this requires interaction ( not always available!) • may override tool NOK/MISS results

10. called: once per hour by cronjob • has update information, knows about its age don’t hesitate to contact QC!

11. documentation • association rules • contact • planned: • include ticket mechanism (PSO scheme) • memory function for long-term calibrations (once a month etc.) and HC calibrations

12. general rule: calChecker follows calOBbuilder • not planned: • short-term night calibrations (like standard stars within 2 hrs) • complex dependencies (multiple standard stars a night; alternative calibrations; calibrations for calibrations) • to be done on Paranal: • optimize calOBbuilder queue (input provided by QC)

13. 3. Links to product quality calChecker HC monitor products/AB monitor • goes beyond completeness, includes quality • links to the QC processing page for that date (AB product monitor)

14. AB product monitor

15. Scores • automatic evaluation scheme: • 2 thresholds defined • QC1 parameter evaluated • total AB score: 0 / 1 / more than 1 • counts outliers

16. links to other reports • data report • nightlog (password protected, sciops)

17. 4. Quick-look HC monitor calChecker HC monitor products/AB monitor • www.eso.org/HC • final QC Garching data now display as quick as/quicker than opslog!(forget 10 day gap…)

18. full version score version (Quick-look)

19. Information on demand:click on “more” if you want graphs, on “less” if you want scores only

20. AB product monitor: all scores of a date • HC monitor: selected scores • selection: “relevant for instrument health”

21. score propagation: • data scores for last week only • plot: last data scores summed upplot score green if last data score is green (previous red scores ignored) • report score: plot scores are aggregated in report score • report scores in group scores, and finally in instrument score • all higher-level scores turn red if at least one lower-level score is red

22. big advantage of score hierarchies: • instrument score green  you know within a second: all is OK! • red: the score overview can tell you more • scheme intended to guide you to issues

23. Health Check monitor • update pattern: refresh most important ones upon new opslog data • less important ones at least once per 24 hour • process health • monitored: opslog transfer, parser (1x per hour) • coming soon: data transfer, ngas access

24. planned: • defocus the OPSLOG data (to become a fallback solution except for VIRCAM/OCAM) • improve update pattern for HC reports (immediate after new data) • product quality links (same as for calChecker) • agree on mail alert mechanism

25. calChecker HC monitor products/AB monitor

26. the end …

27. Other QC services within this context Agora: FAQ page for PSO-QC related issues refs: reference frames web page QC: tutorial pages, documentation

28. avoid being under-sensitive: overlook that raw file is saturated or empty miss important properties = QC1 parameters avoid being too sensitive: have too narrow thresholds have unstable thresholds (3SIG) use not fully understood parameters select data sets (setups) rarely acquired ( trending impossible or uncertain) use same QC1 parameter in various HC plots ( multiple alarms) rule of thumb: every red score should have an explanation likely trigger an action [Excursion: The art of scoring … ] [E. Munch, Cry]

29. 5. Operational impact check Health Check monitor • check scores • pick red ones • analyze problem • trigger actions • daytime astronomer: check completeness • take action for ylw/red boxes: e.g. trigger calibrations check product quality (AB monitor)