ESA Scientific Archives and Virtual Observatory Systems

1. ESA Scientific Archives andVirtual Observatory Systems Christophe.Arviset@esa.int Science Archives and VO Team Research and Scientific Support Department ESA – ESAC – Madrid, Spain

2. European Space Astronomy Centre • ESAC default location for: • Science operations, • long history with astronomical missions, • now expanding with solar system missions. • Science archives, • Astronomy • Planetary • ESA VO activities, • ESAC to be the European VO node for space-based astronomy. http://www.esa.int/SPECIALS/ESAC/ Located near Madrid, Spain

3. Planetary Science Archive Giotto, Mars Express Rosetta, Venus Express Smart-1, Huygens Since March 2004 XMM-Newton Science Archive Since April 2002 ISO Data Archive Since December 1998 Integral SOC Science Data Archive Since July 2005 Herschel, Planck, GAIA, … in the future Archives at ESAC • ESAC is the Centre where most of ESA Scientific Directorate’s Scientific Archives are developed, maintained and operated. • ESAC Science Archives Team is giving support to various projects

4. Archives at ESAC (cont’d) • Scientific Archives geared to various users • Scientific Community (public access) • PI team and observers (controlled access) • Science Operations Team (privilege access) • Common Architecture and Look and Feel • Better corporate image for ESA • Faster, cheaper, better development • Re-use of expertise • User friendly access • Eg PSA Mars Map Browser

5. Science Archives Team @ ESAC • A core Science Archives and VO Team (~15 people) in ESAC • Support many projects (horizontal support vs vertical organization) • ISO, XMM-Newton, Integral, Planetary missions, Herschel, • Virtual Observatory • Soho, Planck, Gaia in the future… • Organized by functions: database, user interface, data distribution, inter-operability • Close collaboration with Archive Scientist (1 per project) • Expert of the science field, close to the community • Consolidate user requirements • Set development priorities • Final acceptance tests • Strong re-use of design/system/code between projects • Not responsible for writing data reduction software (pipeline) nor data production

6. Code re-use across Archives • Manpower shared between projects • Knowledge transfer between projects • Small fraction of FTE for “old” projects • All Archives share a common part of the code • Same functionalities across projects • But still some very specific projects functionalities • Faster and reliable development of new projects • Easier and cheaper maintenance of existing systems • Same Look & Feel for ESA Science Archives User Interface • Powerful and easy to use • Better corporate image for ESA • Easy learning curve for users

7. Data Repository • All Data from all archives are stored on magnetic disks • Standard formats (FITS for astronomy, PDS for planetary) • Save various levels of data • IDA (0.5TB), XSA (3TB), ISDA (6TB), PSA (4TB) • Small amount still • Unique disk repository for proprietary and public data • Access controlled by the application server • Access to data is immediate • Via the Archive User I/F or for the VO • Except for on-the-fly reprocessing requests • Clever FTP server using links instead of real files • Same disk space for FTP • Higher quotas for users

8. DataBase • Strong effort of Data Modeling before starting any project • Unique database for all uplink and downlink metadata • All metadata is systematically extracted from uplink AND downlink data • Done when products are generated • Can be re-run with for new metadata is user reqs change • Sybase for some projects • ISO (10GB), XMM-Newton (6GB), Planetary (2GB) • Oracle for others • Integral (6GB) • PostGres for future projects (Herschel, Soho, …)

9. GUI Database/Product Request DB/Product Request GUI Database Result / Product DB Result / Product Meta Data in Database Product Request Products Project Business Logic Data Products on hard disks Translation Layer Existing Access – GUI • User friendly web access : Java Applet • Powerful queries, organized by panel, hundreds of queriable and displayable parameters • Public data accessible to all, proprietary data accessible only to privilege users • Images preview (icons, full image) • Quick download (1 click) at various processing levels, shopping basket retrieval • Remote access to other archives (ADS, and tools (Aladin, VOSpec) • Project prior to the VO • Existing data model, data format, access mechanisms

10. GUI Database/Product Request DB/Product Request GUI Database Result / Product DB Result / Product Project Product Request DB/Product Request Scriptable Interface Project Product DB Result / Product • foreach obs in (observation_list)        AIOgetfile (obs, file_required)        analyse_process file_required        save results        delete file_requiredend Meta Data in Database Project Business Logic Product Request Products Data Products on hard disks Translation Layer Existing Access : Scriptable I/F • Access to metadata (via XML scripting) • Access to data through socket (direct data stream) or FTP (several files or individual file) • Own Interoperability standards and needs as required by the project • Not using VO protocols, but ready for it…

11. GUI Database/Product Request DB/Product Request GUI Database Result / Product DB Result / Product Project Product Request DB/Product Request Project Product DB Result / Product Scriptable Interface Meta Data in Database • (Scriptable Interface) • foreach obs in (observation_list)        AIOgetfile (obs, file_required)        analyse_process file_required        save results        delete file_requiredend Product Request Products Project Business Logic Project Product Request SIAP/SSA Request DB/Product Request Project Product VOTable / Product DB Result / Product Data Products on hard disks Translation Layer Translation Layer VO compliant Architecture • Existing scriptable I/F already carried all the functionalities required by the new VO standards • Access to metadata (database) • Access to products (data repository) • By using “Translation Layer”, easy to convert the VO standards interfaces into our existing I/F • Little effort required • Translation Layer easy to code using XML • As VO standards evolve, our scriptable I/F system remains, but we just have to adapt our translation layers

12. http://esavo.esac.esa.int/ ESA-VO Main Goals • ESA part of the EURO-VO • ESA part of the IVOA (International VO Alliance) • RSSD astronomical archives being VO compatible • VO in mind when building archives • Archives and VO work done by the same team • Develop VO applications • eg VOSpec, VOQuest, … • Develop VO publishing services • Registry of VO resources, • DALToolkit, DMMapper, … • ESAC to disseminate VO technology • Link to existing and new ESA astronomy projects • VO outreach (workshop, conferences, …)

13. Select Source IUE spectra IUE spectra IUE spectra ISO spectra ISO spectra ISO spectra Find/Select in the VORegistry VO Spectral Servers Display Theoretical Spectra and Observational Spectra Normalize Theoretical Spectra Theoretical Spectra Fits Observational Spectra VEGA Theoretical Model VEGA Theoretical Model VEGA Theoretical Model VO in action : VOSpec application

14. Interoperability ESA PSA – NASA PDS • (Prototype in development) in the context of the IPDA (International Planetary Data Alliance) • From Mars Map Browser, Select region of interest • Contact PSA and PDS using the common data (image) access protocol • Display NASA PDS and ESA PSA images

15. Conclusion • ESA Scientific Archives built and operated within a same team • ESA Archives flexible architecture allows easier and long-term maintenance • ESA strongly believes in the Virtual Observatory, both for astronomy and for planetary missions • Strong link between Archives development and Virtual Observatory work • ESAC is becoming the Data Centre for ESA space-based archives • ESAC is becoming the VO node for European space-based astronomy