Synthesis of LoI's F. Mignard OCA/Cassiopée

1. Synthesis of LoI's F. Mignard OCA/Cassiopée

2. Main Statistics • 162 answers received and validated • 29 data processing in general • 7 for calibration • 10 double and multiple stars • 16 photometry • 10 planetary systems • 13 radial velocity • 8 relativity and reference frame • 5 science alerts • 12 simulations • 17 solar system objects • 20 source classification • 17 variable stars

3. Major Coordinated answers • A set from Cambridge to lead the photometric processing • A combined answer from Cambridge and Geneva for the science alert • One answers from Geneva coordinating the involvement of ISDC • One series from ARI to cover the first look, bright stars analysis • Two coordinated letters from CNES covering support for simulation, major involvement in the data analysis, system architecture • One global answer from Italy for a quality and validation system • An international answer coordinated by Meudon for the RVS analysis • One global answer from the SSWG to carry out the analysis for solar system objects

4. Answers for the Variable stars

5. Synthesis of answers • Number of answers • Collective : 6 • Individual : 11  but some may have considered a personal reply unecessary • Areas covered • Detection of variability • Analysis of light curves • Period determination • Classification • Follow-up • Science alert • Areas marginally covered • Simulation • overall management

6. Answers to LoI : collective answers

7. Answers to LoI : individual answers

8. Related proposals

9. Organisation of the Data Processing F. Mignard From GST and DACC inputs

10. Objectives and challenges • Gaia will observe over 5 years with three instruments • Gaia will produce ~ 200 TB of raw data • At the lowest level they are primarily CCD counts • The DP must cope with • A large treatment both in volume and computation • A complex and iterative treatment • Interconnected data and tasks • It is a community task, neither organised, nor funded by ESA • If must be ready by the launch time

11. Objectives and challenges • It requires specific expertises and commitment : • Overall analysis of a complex project • Professional S/W developers • Quality control • Strong commitment of teams over ~ 10 to 15 years • Ability to work with large team under pressure with tight schedule • The basic organisation does not exist yet • This is very different from the WG structure • The level of coordination is much more demanding • Science will not be the primary activity in the DP • The organisation must have clearly defined responsibilities

12. Overall Concepts • There are three obvious main steps in the DP • The initial data treatment of the CCD images up to centroiding • The astrometric, photometric and spectroscopic iterative solution • The special non iterative processes • The global structure should be • Must be as simple as possible • Clear hierarchy with non overlapping responsibilities • GST must have the full visibility and retain control of the DP • The current workplan • Global system called Gaia Data Analysis Consortium (DAC) • Central concept around Coordination Units (CU) • Within each CU there are Development Units (DU) • The lower level is the Work Package (WP)

13. Coordination Units • Small in number ( < 10) • Each must be in charge of a significant fraction of the DP • Interfaces betweens CU's are limited • Each CU coordinates the developments of many tasks • It defines the structure in DU • It follows the development, testing and documentation of the S/W • It defines and implement the interfaces between the DUs • A CU is placed under the responsibility of a manager and his deputy • They manage their CU • They provide contact and interfaces with the other CUs

14. Identified Cus (provisional) • C1 : Overall system architecture and central DB • C2 : Data simulation ( three levels) • C3 : Astrometric core processing • C4 : Object processing • C5 : Photometric reduction • C6 : Spectroscopic reduction • C7 : [Scientific exploration and Catalog access]

15. Identified Cus leaders and deputies • C1 : Overall system architectureW. O'Mullane, U. Lammers • C2 : Data simulation X. Luri, C. Babusiaux, F. Mignard, • C3 : Astrometric core processingU. Bastian, M. Lattanzi, J. Torra • C4 : Object processingD. Pourbaix, C. Bailer Jones, P. Tanga • C5 : Photometric reductionF. van Leeuwen, A. Brown, C. Jordi • C6 : Spectroscopic reductionD. Katz, M. Cropper, U. Munari

16. Developpement Units • Responsibility for the development of S/W • Conceptual development • Algorithm testing on simulated data • Preparation of the implementation of the operational versions • Process the Gaia data and provide outputs for the DB • A DU should be based on a small number of well connected groups • Typically 5 people in each place • Examples of Dus : • PSF calibration, RV determination, orbit determination, taxonomic classification

17. Architecture and H/W • Overall architecture not yet defined • Several approaches possible : • All computational H/W and DB at one place  serious funding problem • All core processing at one place + distributed shell tasks  data transfer • Centralised data base and core tasks ; separate location for integration of shell tasks and centralised or distributed execution • Local data bases with shell tasks processing and coordination units

18. 11h mean GS visibility per day Astro-1: 20 TB Astro-2: 20 TB RVS : 30 TB MBP : 25 TB MOC 50 GB/day 5 Mbps downlink rate TM Archive GAIA primary Data Base C2 Initial treatement Calibrations First look System determination (GIS) C3 Simulation Data Streams Images C5 C6 C4 Photometry Spectro Object Photo DB Spectro DB Object DB Class. DB Objects MBP BBP Variability Alerts Calibrations SGIS RV AP DMS Solar System Planets Galaxies QSOs AP Age Global Class.

19. 11h mean GS visibility per day Astro-1: 20 TB Astro-2: 20 TB RVS : 30 TB MBP : 25 TB MOC 50 GB/day 5 Mbps downlink rate TM Archive GAIA primary Data Base ESAC, UB , ISDC ? C2 Initial treatement Calibrations First look System determination (GIS) C3 Simulation Data Streams Images C5 C6 C4 Photometry Spectro Object Photo DB Spectro DB Object DB Class. DB Objects MBP BBP Variability Alerts Calibrations SGIS RV AP DMS Solar System Planets Galaxies QSOs AP Age Global Class. UB,CNES Cambridge Genève CNES

20. How to form the Consortium ? • Setting of a DACC (Data Analysis Coordination Committee) • Must establish the structure of the consortium : • Breakdown into CU compatible with data flow • List of DUs within each CU • Examine the funding issues, data flow problem, system architecture • Identify institutes, teams or individuals for the WPs • DACC established for one year • Will be replaced by the DACE (Data Analysis Consortium Executive) • An AO will be issued to formalise the structure • no competitive AO is foreseen

21. Composition of the DACC • Chair : F. Mignard • Co-chair : C. Bailer-Jones • CU managers : • W. O'Mullane, X. Luri, U. Bastian, D. Pourbaix, F. Van Leeuwen, D. Katz • Project scientists : M. Perryman • + Experts : A. Brown, U. Lammers, L. Lindegren, J. Torra

22. Pyramidal structure • GST (includes all the CU managers) • Data Analysis Consortium Executive (DACE) • Consists of the CU managers • Coordination Units • Manager, deputy manager, S/W engineers, DUs • Development Units • Leader, includes several work packages • Work Packages • From individual to small groups within one location

23. Next Steps • List of work-packages of each CU • Compilation document in preparation • Construct the data flow constraints • Functional analysis • List of processes and dependencies • CU managers build their architecture of DUs • Should reflect the answers to the LoI • Large contact with the community needed • Next meeting of the DACC : 6-7/10 in Heidelberg

24. Gaia Data Flow, V1.1, 20-6-05 (MP, WOM, UL, LL, JdB +UB, FvL) MOC: Telemetry Archive Housekeeping, Orbit Files Operations Reports Heidelberg Pass Archive (flat files, t order) Real-Time Attitude (~6 arcsec) Image parameters Initial cross-matching On-ground att. 50 mas Decoding Decompression Gaia Main Data Base Cross Matching SAGC VN-1 Source (+RV) Attitude Calibration Global Source (+RV) Attitude Calibration Global INTEGRATOR SGC VN-1 PSF/LSF calibration CCD calibration A/P/S calibration First Look VN AC new VN-1 Cross Matching (A/P/S) Raw Data Base [Oracle] 1 iteration earlier Spatial Ordering (HTM) Element Astrom Element Photom Match Table Element Astrom Element Photom Match Table Science Alerts Update LSF centroid Position + Flux ESAC Construct and apply Pre-pass Catalogue (VN-1) ESAC ESAC ASPC results communicated via ID + parameters + data base flags + other flags Heidelberg, Bruxelles, Paris Lund, Barcelona Cambridge, Leiden, Geneva Meudon, MSSL Object Tasks & Classification + Data Base Astrometry + Data Base (sample tasks in GDB) Photometry + Data Base (position and flux information) (sample tasks in GDB) Spectroscopy + Data Base (sample tasks TBD) Astrometric Global Iterative Solution Source Attitude Calibration Global Source Updating (secondary stars) Photometric Calibration Parameters Photometric Calibration Model Photometric Calibration Stars Cross Correlation CCD/FoV/Mission Spectroscopic Global Iterative Solution Calibration Visual binaries Astrometric binaries Planets Solar system objects ASM imaging (disturbing stars) Calibration Monitoring Reduced photometric data Constant stars Radial velocities Calibrated spectra Metallicity, Rotation Source classification Astrophysical parameters Reference Frame Epoch photometry Accumulating statistics Variability analysis CNES? CNES? ESAC Unexpected change in brightness/classification Period search Period type? Alert filter Period type? 18 0 6 12 months Alerts: SNe, NEO Yes: Periodic variables No: Unsolved variables V2 Integrator V0 (ground) V1 Timeline 8-1 weeks GIS + shell Cambridge ?