Computational and statistical problems for the Virtual Observatory

1. Computational and statistical problems for the Virtual Observatory With contributions from/thanks to: GAVO team: Wolfgang Voges, Matthias Steinmetz, Harry Enke, Hans-Martin Adorf Joerg Colberg (NVO@UPitt), Pat Dowler (CVO), Tony Banday (MPA), Class X team CMU-CS lunch talk, Gerard Lemson

2. Overview • Intro to VO • IVOA standards process • Some concrete examples, demos • Scenarios, science cases • Interesting problems CMU-CS lunch talk, Gerard Lemson

3. Intro to VO • Very large data sets • Multi-wavelength astronomy made easy • Federation of distributed archives. • Publication of expert services. • New software developments. • Why contribute ? • Too easy to do bad science ? CMU-CS lunch talk, Gerard Lemson

4. CMU-CS lunch talk, Gerard Lemson

5. IVOA standards and specifications • Collaboration of national VOs • Develop standards for interoperability • publication (registry) • description (dm, ucd) • query (dal, voql) • data transfer (votable) • services (grid/web services) • Interest groups: • architecture • applications • theory CMU-CS lunch talk, Gerard Lemson

6. Babylonian confusion CMU-CS lunch talk, Gerard Lemson

7. VO domain model as Esperanto CMU-CS lunch talk, Gerard Lemson

8. CMU-CS lunch talk, Gerard Lemson

9. CMU-CS lunch talk, Gerard Lemson

10. Protocols • VOTable + UCD  DM based XML + XSLT • SCS/SIAP/SSAP  ADQL  VOQL • SkyNode • Registry resource model and harvesting interface CMU-CS lunch talk, Gerard Lemson

11. Data models • Targeted “small” data models • Quantity • Observation • Simulation • Domain model as ontology • Meta-data repository • Bindings • Representations, views, transformations CMU-CS lunch talk, Gerard Lemson

12. CMU-CS lunch talk, Gerard Lemson

13. CMU-CS lunch talk, Gerard Lemson

14. Theory in the VOWith Joerg Colberghttp://ivoa.net/pub/papers/TheoryInTheVO.pdf • Spatial query protocols irrelevant • No object-based federation • New phenomena/observables. • Different kind of provenance. • Model dependency. • Theoretical archives rather unstructured. • Theory/observational interface. CMU-CS lunch talk, Gerard Lemson

15. Observed Simulated Thanks to Volker Springel Thanks to Alexis Finoguenov, Ulrich Briel, Peter Schuecker, MPE) CMU-CS lunch talk, Gerard Lemson

16. Some concrete efforts • NVO (USA): Registry (DIS), ADQL, SkyNode, data mining (UPitt+CMU) • AstroGrid (UK): grid/web services, work flows • AVO (ESO, CDS, AstroGrid): Aladin visualization tool, science demos • CVO (Canada): archive federation • France VO: GalICS • GAVO (Germany): data publication (RASS photons), application prototypes, data mining, theory CMU-CS lunch talk, Gerard Lemson

17. Scenarios, use cases, results • Registry based data discovery and retrieval (GAVO, DIS) • Class X classifier and generalizations • X-Ray cluster analysis using simulations • Cluster detection by combining SDSS and RASS catalogues (Schuecker et al, astro-ph/0403116) • Discovery of obscured quasars using VO tools (Padovani et al, astro-ph/0406056) CMU-CS lunch talk, Gerard Lemson

18. Typical workflow CMU-CS lunch talk, Gerard Lemson

19. Download manager CMU-CS lunch talk, Gerard Lemson

20. ClassX@GAVO CMU-CS lunch talk, Gerard Lemson

21. Theory/observational interface: X-Ray clusters Goal: interpret observations of X-Ray cluster using results of hydro simulations: • Extract parameters from the observation (services) that can be queried directly (dm, ucd). • Find simulations that may be relevant, that are “similar” to observation by searching registry for hydro simulations of clusters (registry, voql). Requires simulation results to be published and described in sufficient detail (dm, ucd). • Observe simulations using “virtual telescope” (application, grid/webservices) configured according to telescope configuration extracted from observation (dm). • Compare real with virtual observation (services). • For interesting simulation, extract full simulation result (dal) for further analysis, • or analyse the simulation using services (grid-services) provided by the archive or some other service provider CMU-CS lunch talk, Gerard Lemson

22. CMU-CS lunch talk, Gerard Lemson

23. Computational, statistical and astronomical challenges I Data models • Data modeling • Data model transformations, views • Archive structure • Database tuning Querying, matching • Distributed query algorithms • Probabilistic matchers, systematic errors, identification of moving sources • Improve identification using full point process information • Add physical properties, not just position, to identification • Complex, frequency dependent source definition • Characterization of complex results in "few" parameters for discovery (PCA (after transformation)? 3D->2D ?) • Comparison of real and virtual observations CMU-CS lunch talk, Gerard Lemson

24. Usage • Complex model • Simplify using view concept • Example from RDB • XSLT for translation between domain XSD and application-specific derived schemas. CMU-CS lunch talk, Gerard Lemson

25. CREATE VIEW SEXTRACTOR_GALAXIES AS SELECT S.RA AS _RAJ2000, S.DEC AS _DECJ2000, -2.5 * LOG(S.FLUX) AS M_APP, S.CLASSIFICATION, I.STORAGE_URL AS IMAGE FROM SOURCE S, SOURCE_CATALOGUE SC, IMAGE I, SOURCE_EXTRACTOR AS SE WHERE S.CLASS = ‘GALAXY’ AND S.FLUX < 15 AND S.CATALOGUE_ID = SC.ID AND IMAGE.ID = SC.IMAGE_ID AND SC.EXTRACTED_WITH = SOURCE_EXTRACTOR.ID AND SE.IDENTIFIER = ‘SExtractor’ CMU-CS lunch talk, Gerard Lemson

26. Probabilistic cross matching CMU-CS lunch talk, Gerard Lemson

27. Computational, statistical and astronomical challenges II Data mining • Algorithms for analyzing generic SEDs (classifiers ? visualization ? incorrect identification ?) • Source extraction using multiple images, at very different wavelengths, how to take into account different physics/images of same source at different wavelengths ? • Cluster finders using multiple catalogues • Publish sophisticated statistical analysis algorithms Implementation • Efficient implementation virtual telescopes (parallel, distributed, grid based, data structures) CMU-CS lunch talk, Gerard Lemson