390 likes | 547 Vues
Euro-VO DCA WP4 and GAVO and (a little) AstroGrid-D. Gerard Lemson Euro-VO DCA WP5 kick-off meeting Trieste, 2006-11-30. Overview. Euro-VO DCA WP4 Goal and motivation Theory in VO Theory in IVOA Theory in Euro-VO DCA: WP4 WP4 and WP5: theory WP5 and/or grid in WP4 ?
E N D
Euro-VO DCA WP4andGAVO and (a little) AstroGrid-D Gerard Lemson Euro-VO DCA WP5 kick-off meeting Trieste, 2006-11-30 Euro-VO DCA WP5 kick off
Overview • Euro-VO DCA WP4 • Goal and motivation • Theory in VO • Theory in IVOA • Theory in Euro-VO DCA: WP4 • WP4 and WP5: theory WP5 and/or grid in WP4 ? • GAVO (& AstroGrid-D) • GAVO I • GAVO II • Theory in GAVO • Grid in GAVO • AstroGrid-D • (GA)VO and AstroGrid-D Euro-VO DCA WP5 kick off
Euro-VO DCA WP4 • Objectives • This task aims at assessing the inclusion of theory data and services in the Virtual Observatory • Description of work • The task is tackled by the Theory Expert Group, which has yearly meetings and email discussions to establish a framework for the inclusion of Theoretical Astronomy Services in the VObs. The TEG organizes a Workshop. • Deliverables • WP4-D10: Euro-VO Theory Workshop • WP4-D11: TEG Report: Framework for inclusion of Theoretical Astronomy Services in the VObs • Expected results • The final report will describe a framework for inclusion of theoretical data and services in the VObs • Motivation of the European Theoretical astronomy community to participate in the VObs endeavour. Euro-VO DCA WP5 kick off
Reminder • Broad goal of VObs • Make results of astronomical research, data and applications, more readily available to larger community, and create value-adding services. • Facilitate results- • communication • checking • (re)use • comparison • combination • Organised in IVOA Euro-VO DCA WP5 kick off
The observational VObs • Most VO efforts concentrate on observational data sets • simple observables: photons detected at a certain time from a certain area on the sky • long history of archiving • pre-existing standards (FITS) • valuable over long time (digitising 80 yr old plates) • long history of comparing different observations: interoperability • Standards observationally biased • common sky: cone search, SIAP, region • common objects: XMatch • data models: characterisation of sky/time/energy (/no polarisation yet) Euro-VO DCA WP5 kick off
The same object X-Ray Radio Optical John Hibbard http://www.cv.nrao.edu/~jhibbard/n4038/n4038.html NASA/CXC/SAO/G. Fabbiano et al. Euro-VO DCA WP5 kick off
Theory in VO • Apply same ideas to theoretical resources (datasets and services). • Plenty of resources available online. • Not standardised: • 2004, whitepaper detailing issues related to standardisation of theory Euro-VO DCA WP5 kick off
Theory in the VO: issues • Simulations not so simple • complex observables • no standardisation (not even HDF5) • archiving ad hoc, for local use • Moore’s law makes useful lifetime relatively short: few years later can do better Euro-VO DCA WP5 kick off
“Moore’s law” for N-body simulations Euro-VO DCA WP5 kick off Courtesy Simon White
Standardisation issues • Current IVOA standards somewhat irrelevant • assume common sky • common objects • comparison direct • In theory • no common sky • no common objects • comparison statistical • requires new data models • content, physics, code Euro-VO DCA WP5 kick off
Not the same object Toomre & Toomre, 1972 Courtesy Volker Springel Di Matteo, Springel and Hernquist, 2005 Euro-VO DCA WP5 kick off
So why bother ? • Simulations are interesting: • For many cases only way to see processes in action • Complex observations require sophisticated models for interpretation • Bridging gap in specialisations: not everyone has required expertise to create simulations, though they can analyse them. • Many use cases do not require the latest/greatest • exposure time calculator • survey design Euro-VO DCA WP5 kick off
Theory in the IVOA • IVOA Interest group on theory • founded May 2004 • “horizontal” • use cases • Current projects: • Theory UCDs/Semantics (Laurie Shaw) • SNAP (Claudio Gheller) • Simulation (meta-) data model (GL) • Meeting on SNAP in Garching, early 2007 Euro-VO DCA WP5 kick off
Euro-VO DCA WP4 (repeat) • Objectives • This task aims at assessing the inclusion of theory data and services in the Virtual Observatory • Description of work • The task is tackled by the Theory Expert Group, which has yearly meetings and email discussions to establish a framework for the inclusion of Theoretical Astronomy Services in the VObs. The TEG organizes a Workshop. • Deliverables • WP4-D10: Euro-VO Theory Workshop • WP4-D11: TEG Report: Framework for inclusion of Theoretical Astronomy Services in the VObs • Expected results • The final report will describe a framework for inclusion of theoretical data and services in the VObs • Motivation of the European Theoretical astronomy community to participate in the VObs endeavour. Euro-VO DCA WP5 kick off
WP4: tasks • Procurement WP4 personnel: • Draft whitepaper “Framework for inclusion of Theory data and services in the VObs” (End cycle 1) • EURO-VO Theory Workshop (Deliverable 10, Month 17) • Whitepaper ready (Deliverable 11, month 26) • Participation IVOA interop meetings with theory Euro-VO DCA WP5 kick off
WP4 first steps • Formation Theoretical astronomy Experts Group (TEG) • CNRS Hervé WOZNIAK • ESA Pedro OSUNA • ESO None • INAF Santi CASSISI • INTA Miguel CERVIÑO • MPG Gerard LEMSON (Chair) • NOVA TBD (Joop SCHAYE) • LU TBD (?) • Census theory data centers (with WP2) • Cooperation in IVOA theory interest group activities • Coordination with WP 5 (which is why I’m here) Euro-VO DCA WP5 kick off
WP4 and WP5: some first ideas • Grid support for WP4 • Grid must be addressed in WP4 whitepaper • Grid can facilitate/is required for implementations of prototype for standards (see below) • Theory use cases/requirements for WP5. • Asynchronous, queued querying of large simulation database • OGSA-DAI? (GAVO+AstroGrid-D) • SNAP on Millennium raw data • grid services wrapping custom data access modules (GAVO+AstroGrid-D) • Scientifically useful virtual telescopes • Parameter studies using “small” simulations • Large simulations ?? Euro-VO DCA WP5 kick off
Example: Virtual Telescope Observed Simulated Thanks to Volker Springel Thanks to Alexis Finoguenov, Ulrich Briel, Peter Schuecker, MPE) Euro-VO DCA WP5 kick off
Virtual telescope configuration Euro-VO DCA WP5 kick off
Virtual telescope configuration • Components: • simulation of source, wrapped with “photon producer” service • spec and implementation for “pre-observation image” (POI) • “foreground” accepting, reemitting POI • ISM/IGM • lenses • “software telescope”, simulating realistic observations • comparison software • Required (VO/Grid) infrastructure supporting this workflow • registry for publishing/discovering components • computational resources • scheduler for executing job • streaming support for flowing POI between components • storage for results • workflow engine? Euro-VO DCA WP5 kick off
Discussion Suggestions very welcome Euro-VO DCA WP5 kick off
Intermezzo Euro-VO DCA WP5 kick off
GAVO and AstroGrid-D Euro-VO DCA WP5 kick off
GAVO and AstroGrid-D • GAVO I • Funded by German Ministry of • Dec 2002 – March 2005 • 3 months extension • 3 FTEs • AstroGrid-D • 1 of 5 funded “communities” taking part in D-Grid project • O(13) FTEs • Autumn 2005 – 2007/8 • GAVO II • Delayed funding, due to need to distinguish from AstroGrid-D • June 2006-July 2008 • 5 FTEs (so far 3, ARI (>~1.5), MPE (1.5), TUM (.5), AIP (<~.5), Tübingen (1)) Euro-VO DCA WP5 kick off
GAVO (I + II) projects • Archives: • ROSAT All Sky Survey • cone search: photons, sources • SIAP: fields • ROSAT pointed observations: SIAP • standardised approach to publish FITS images under SIAP • uses FITS metadata repository (SQLServer) • SQL based mapping and SIAP implementation • easily generalised to other datasets and protocols (SSA) • RAVE • CDFS optical follow up spectroscopy: SSA • Heidelberg catalogues and GAVO archive: SCS/SkyNode • Data mining • Cross-matcher (also on Grid, TUM) • SED classifier • Cluster finder Euro-VO DCA WP5 kick off
GAVO (I + II) projects ctd. • Grid • evaluation Globus • Gadget 2 on Globus MPI (AIP + MPE) • cluster finder on the grid • VOQL/SkyNode • VOQL tiger team (Benjamin Gufler) • HiSbase peer-to-peer approach to multi catalogue cross-matching • Mock catalogues for testing • Theory • IVOA: • Initialised Theory Interest Group (chair) • SNAP • Data models • Semantics • Millennium database • (proto-) Virtual telescopes (planck, MoMaF2, Dolag et al) Euro-VO DCA WP5 kick off
AstroGrid-D • Part of D-Grid • Community project • Provide • use cases • middleware • adaptation of existing applications/data to the (D-)Grid Euro-VO DCA WP5 kick off
AstroGrid-D partner projects (incomplete) • Astrophysikalisches Institut Potsdam • project management • robotic telescopes on the grid • galaxy merger simulations • MPE: • cluster finder • catalogue cross-matcher (with TUM) • adapt simulation visualisation to the Grid • access to persistent data (in database) • MPA • Planck Process Coordinator on the Grid • Technischer Universitat München-Informatik • StarGlobe: streaming data querying/combination Euro-VO DCA WP5 kick off
AstroGrid-D partner projects, ctd. • Zuse Institut Berlin • Grid Application Toolkit • GridSphere web portal • Information Services (“registry”, RDF+SparQL) • Albert Einstein Institute, Berlin • Cactus tool kit • Grid based simulation monitoring • Astronomisches Rechen Zentrum, Heidelberg • Stellar dynamics: Grape on the Grid • GaiaGrid • Planck ProC collaboration Euro-VO DCA WP5 kick off
GAVO + AstroGrid-D, synergies • Groups involved in both projects: AIP, ARI, MPE, TUM • VO offers different kind of use cases: • VObs aims to enable external users to use resources • Heterogeneous world: Interoperate with resources outside of AstroGrid-D (but within IVOA standards) • Grid can help implement VO services Euro-VO DCA WP5 kick off
GAVO + AstroGrid-D ctd. • Planned (from GAVO point of view) • Cluster finder (MPE) • SNAP implementation (MPA) • OGSA-DAI (GSM?, DSA?, CAS Jobs) access to Millennium DB (MPA) • Real virtual telescopes (MPA+...) • Cross matching (MPE+TUM) • Possibly • VOEvent and Robotic Telesopes ? • AstroGrid-D Information Services as IVOA Registry Euro-VO DCA WP5 kick off
SNAP using grid services • Simple Numerical Access Protocol • Project of IVOA theory interest group • Based on simulation data model (input welcome !) • Query interesting simulations • Select and retrieve subset • Implementation: SNAP on Millennium snapshots • Custom access code deployed near large datasets • Authenticated access • Standardised publication for use in Grid workflows (ProC? Streaming?) • Data staging and movement Euro-VO DCA WP5 kick off
Globus container SNAP web service Grid Service getBox() C-module Information Service Millennium snaphots IS aware Workflow Engine Euro-VO DCA WP5 kick off
Millennium website and OGSA-DAI • Dataset • postprocessing products of Millennium simulation (http://www.mpa-garching.pg.de/millennum) • halo/galaxy catalogues • density fields (environment queries) • Functions • support generic SQL queries on large (O(5Tb)), complex database • support mydb • some graphics script based access Euro-VO DCA WP5 kick off
Implementation • Microsoft SQLServer database (SkyServer inspired) • data model for efficient retrieval merger tree histories • spatial indexing • TOMCAT web server • http://www.g-vo.org/MyMillennium • authentication using RDB • wget based access, IDL+R modules • Graphics • VOPlot applet (small results, browser) • TOPCAT (plugin) • Synchronous queries • JDBC • 7 min time limit • Issues • I/O bottle necks when > 4 users query same table • short maximum query time • no asynchronous access • simplistic user management (compared say to CAS Jobs) Euro-VO DCA WP5 kick off
Why Grid interesting? • Need • Asynchronous access • Queuing • Improve result management, data delivery • Bulk uploading into MyDB • More generic authentication (X.509) • Will try CAS jobs (SDSS/JHU) • Investigate OGSA-DAI • Map portal to OGSA-DAI functionality • Monitoring ? Logging ? Performance ? • Will try AstroGrid-UK’s DSA • Other suggestions ? Euro-VO DCA WP5 kick off
Thank you ! Euro-VO DCA WP5 kick off