Alex Szalay Department of Physics and Astronomy The Johns Hopkins University

1. New Astronomy from The Sloan Digital Sky Survey to the National Virtual Observatory Alex Szalay Department of Physics and Astronomy The Johns Hopkins University and the SDSS Collaboration

2. SDSS: Distributed Collaboration Fermilab U.Chicago U.Washington ESNET I. AdvancedStudy Japan Princeton U. VBNS JHU Apache PointObservatory NMSU USNO EU/US Workshop, Alex Szalay

3. Scientific Motivation Create the ultimate map of the Universe: The Cosmic Genome Project! Study the distribution of galaxies: What is the origin of fluctuations?  What is the topology of the distribution? Measure the global properties of the Universe: How much dark matter is there? Local census of the galaxy population: How did galaxies form? Find the most distant objects in the Universe: What are the highest quasar redshifts? EU/US Workshop, Alex Szalay

4. Features of the SDSS Special 2.5m telescope, located at Apache Point, NM 3 degree field of view. Zero distortion focal plane. Two surveys in one: Photometric survey in 5 bands. Spectroscopic redshift survey. Huge CCD Mosaic 30 CCDs 2K x 2K (imaging) 22 CCDs 2K x 400 (astrometry) Two high resolution spectrographs 2 x 320 fibers, with 3 arcsec diameter. R=2000 resolution with 4096 pixels. Spectral coverage from 3900Å to 9200Å. Automated data reduction Over 100 man-years of development effort. (Fermilab + collaboration scientists) Very high data volume Expect over 40 TB of raw data. About 1 TB processed catalogs. Data made available to the public. EU/US Workshop, Alex Szalay

5. The First Stripes Camera: 5 color imaging of >100 square degrees Multiple scans across the same fields Photometric limits as expected EU/US Workshop, Alex Szalay

6. SDSS Data Products Object catalog 400 GB parameters of >108 objects Redshift Catalog 1 GB parameters of 106 objects Atlas Images 1.5 TB 5 color cutouts of >108 objects Spectra 60 GB in a one-dimensional form Derived Catalogs 20 GB - clusters - QSO absorption lines 4x4 Pixel All-Sky Map 60 GB heavily compressed Total 2 TB All raw data (40 TB) saved in a tape vault at Fermilab EU/US Workshop, Alex Szalay

7. Distributed Archive Implementation User Interface Analysis Engine Master SX Engine Objectivity Federation Objectivity Slave Slave Slave Objectivity Slave Objectivity Objectivity RAID Objectivity RAID RAID RAID EU/US Workshop, Alex Szalay

8. The Age of Mega-Surveys • The next generation of astronomical archives with Terabyte catalogs will dramatically change astronomy • top-down design • large sky coverage • built on sound statistical plans • uniform, homogeneous, well calibrated • well controlled and documented systematics • The technology to acquire, store and index the data is here - we are riding Moore’s Law • Multi-wavelength view of the sky • - more than 13 wavelength coverage within 5 years • Data mining in such vast archives will be a challenge, • but possibilities are quite unimaginable • Integrating these archives into a single entity is a • project for the whole community=> National Virtual Observatory MACHO 2MASS SDSS DPOSS GSC-II COBE MAP NVSS FIRST GALEX ROSAT OGLE, ... EU/US Workshop, Alex Szalay

9. The National Virtual Observatory National • distributed in scope across institutions and agencies • available to all astronomers and the public Virtual • not tied to a single “brick-and-mortar” location • supports astronomical “observations” and discoveries via remote access to digital representations of the sky Observatory • general purpose • access to large areas of the sky at multiple wavelengths • supports a wide range of astronomical explorations • enables discovery via new computational tools EU/US Workshop, Alex Szalay

10. Trends in Astronomy Future dominated by detector improvements • Moore’s Law growth in CCD capabilities • Gigapixel arrays on the horizon • Improvements in computing and storage will track growth in data volume • Investment in software is critical, and growing Total area of 3m+ telescopes in the world in m2, total number of CCD pixels in Megapix, as a function of time. Growth over 25 years is a factor of 30 in glass, 3000 in pixels. EU/US Workshop, Alex Szalay

11. New Astronomy – Different! Systematic Data Exploration will have a central role in the New Astronomy Digital Archives of the Sky will be the main access to data Data “Avalanche” the flood of Terabytes of data is already happening, whether we like it or not! Transition to the new may be organized or chaotic EU/US Workshop, Alex Szalay

12. NVO: The Challenges Size of the archived data • 40,000 square degrees is 2 trillion pixels • One band: 4 Terabytes • Multi-wavelength: 10-100 Terabytes • Time/spectral dimension: few Petabytes The development of • new archival methods • new analysis tools • new standards (metadata, interchange formats) Hardware/networking requirements Training the next generation! EU/US Workshop, Alex Szalay

13. New Hardware Requirements Large distributed database engines with Gbyte/s aggregate I/O speed High speed (>10 Gbits/s) backbones cross-connecting the major archives Scalable computing environmentwith hundreds of CPUs for statistical analysis and discovery EU/US Workshop, Alex Szalay

14. NVO Development Functions Software development • query generation/optimization, software agents, user interfaces, discovery tools, visualization tools Standards development • Meta-data, meta-services, streaming formats, object relationships, object attributes,... Infrastructure development • archival storage systems, query engines, compute servers, high speed connections of main centers Train the Next Generation • train scientists equally at home in astronomy and modern computer science, statistics, visualization EU/US Workshop, Alex Szalay

15. Issues for EU/US Collaboration Similar efforts at • CDS (see Ochsenbein presentation) • ESTEC (Planck, GAIA), UK (Vista) Expand the NVO to be an international effort Joint development of metadata standards Novel user interfaces for data exploration Comparison of large data sets to large simulations EU/US Workshop, Alex Szalay