Cooking with Sloan: Special DR6 Recipes

1. Cooking with Sloan:Special DR6 Recipes Jordan Raddick, Ani Thakar Alex Szalay, Maria Nieto-Santisteban, Nolan Li, Wil O’Mullane, Adrian Pope, Tamas Budavari, George Fekete, Sam Carliles JHU Brian Yanny, Svetlana Lebedeva FermiLab Jim Gray Microsoft Research

2. Outline • SDSS-II and DR6 • CAS Data Access • SkyServer, ImgCutout, CasJobs • Help resources and sample queries • Restricted collab access • SDSS and other datasets • VO services • EPO content SDSS catalog data demo, March 30, 2007

3. SDSS-II • Legacy • Continuation of SDSS-I (fill out 10k sq.deg.) • Completeness is same as for SDSS-I • Flux limits are the same • Target all galaxies with r_petro < 17.77, plus LRGs • SEGUE • Detailed 3-d map of the Galaxy • Spectra of 240,000 stars in the disk and spheroid • Age, composition and phase space distribution • CAS component cataloged in SegueDRx DB • Supernova Survey • Repeated scans of SDSS Southern Stripe over 3 mths/yr • Data not available in CAS yet, will be on DAS soon SDSS catalog data demo, March 30, 2007

4. SDSS Data Overview • Catalog Archive Server (CAS) • Science parameters extracted to catalogs • Stuffed into relational DBMS (SQL Server) • Heavily indexed, optimized • Online access via SkyServer • Several levels of access, query tools Data Archive Server (DAS) FITS files (raw data) Images, spectra, corrected frames, atlas images, binned images, masks Online form-based access Rsync and wget file retrieval SDSS Data Release cas.sdss.org/collab/ skyserver.sdss.org www.sdss.org das.sdss.org/DRx-cgi-bin/DAS SDSS catalog data demo, March 30, 2007

5. CAS Datasets (databases) • BestDRx • Latest, greatest calibration of the data • Photometric and spectroscopic objects • The default and most accessed (by far) dataset • TargDRx • The calibration from which spectroscopic targets were chosen • RUNS • All the runs (processings) other than Best, Target • SegueDRx • New with DR6 (SDSS-II) SDSS catalog data demo, March 30, 2007

6. CAS Data Access • SkyServer: http://cas.sdss.org/collabdr6 • Also pwd access at http://cas.sdss.org/collabdr6pw for non-collab IPs • See message sdss-archive/2935 for uname/pwd • CasJobs • DR6 and DR6QA targets point to BestDR6 DB • TARGDR6, SEGUEDR6 targets • Need to have “collab” privilege set in user profile SDSS catalog data demo, March 30, 2007

7. CAS Data Access • SkyServer • Synchronous access – need only web browser • Meant to support several levels of users • Visual tools to browse image and catalog data • API access, e.g. emacs interface, sqlcl (command-line) • Strict limits on execution time and output size(but higher for collaboration) SDSS catalog data demo, March 30, 2007

8. Navigating SkyServer • Major sections linked from main page SDSS catalog data demo, March 30, 2007

9. Finding your favorite object • Get position from SIMBAD SDSS catalog data demo, March 30, 2007

10. Finding your favorite object • Advanced Tools -> Navigate • Enter RA/Dec in DMS or HH:MM:SS SDSS catalog data demo, March 30, 2007

11. Finding your favorite object • Click to get position, magnitude • Notebook “shopping cart” SDSS catalog data demo, March 30, 2007

12. Finding your favorite object • Explore – full SDSS data for object • Chart – create finding chart SDSS catalog data demo, March 30, 2007

13. Searching the data • Search for data that meets certain criteria • Return matching data, and only matching data • Example: search for galaxies with spectra in Abell 168 • RA = 18.84, Dec = 0.26, radius = 3 arcmin • Limit to galaxies g < 18 • Return RA, Dec, magnitudes, redshift SDSS catalog data demo, March 30, 2007

14. Searching the Data • Tools ->ImagingQuery • Get data asHTML or CSV • Constraintsby positionor imaging SDSS catalog data demo, March 30, 2007

15. SQL • SQL – computer language for searching databases • Advantages • You get longer queries and more results returned • You can write more complex queries • You can use command-line tools SDSS catalog data demo, March 30, 2007

16. SQL Concepts • Data are stored in a database • Similar data types are stored in tables • photoObj (photometry), specObj (spectroscopy), etc. • A small part of the photoObj table: SDSS catalog data demo, March 30, 2007

17. SQL Concepts • Within a table: • Horizontal rows are individual data points, or records • Vertical columns are types of data, or columns • A request to a database to return data is called a query • Queries usually request data that meets certain constraints SDSS catalog data demo, March 30, 2007

18. SQL as a foreign language • Languages have grammar and vocabulary • Dutch grammar • In sub-clause, verb goes at the end • English: • I think that astronomy is fun. • Dutch: • Ik denk dat het sterrenkunde leuk is. SDSS catalog data demo, March 30, 2007

19. SQL Grammar • Select – choose which columns of data you want to see • From – choose the table(s) from which you want to retrieve data • Where – set constraints on the search SDSS catalog data demo, March 30, 2007

20. Dutch vocabulary SDSS catalog data demo, March 30, 2007

21. SQL vocabulary • Go to Help -> Schema Browser SDSS catalog data demo, March 30, 2007

22. Learning SQL • Go to Help -> Sample SQL Queries • Go to Tools -> Search -> SQL • Copy, paste, modify SDSS catalog data demo, March 30, 2007

23. Visual search results • Go to Advanced Tools ->Image List • Use query to fill form • Query must have:select x, ra, dec SDSS catalog data demo, March 30, 2007

24. CasJobs • Go to Search Tools -> CasJobs • Best method for long or complex queries SDSS catalog data demo, March 30, 2007

25. CasJobs • Register and log in • Quickmode: 1 minute • Submit: 24 hours • Results stored in MyDB SDSS catalog data demo, March 30, 2007

26. CasJobs: other features • MyDB database to save results of your queries • Define your own functions, procedures too • Share your tables with collaborators (groups) • History of • Output data as FITS/CSV/VOTable • Plot data in browser • Table Import (upload) for your own data • Groups to share your results with collaborators • Command-line access Java tool also downloadable SDSS catalog data demo, March 30, 2007

27. Collab-only access • collabdrx SkyServer sites • http://cas.sdss.org/collabdrx • http://cas.sdss.org/collabdrxpw • collab privilege in CasJobs • Restricted data, longer queues (DR6QA 24-hr queue) • If you have collab set, you will see these queues • If you don’t have it, email sdss-helpdesk@fnal.gov SDSS catalog data demo, March 30, 2007

28. SkyServer help • Schema Browser • FAQ • Glossary – of SDSS terms • Algorithms • Helpdesk SDSS catalog data demo, March 30, 2007

29. CasJobs help • Shares SQL Intro, Schema Browser with SkyServer • Has its own FAQ page • Lists differences between CasJobs and SkyServer due to distributed query execution • Advanced CasJobs Queries page • Neighbor searches with fixed and variable search radii • Cursors • Compound queries SDSS catalog data demo, March 30, 2007

30. Find the right tool for the job • When you want to browse images, or when you know a position… • ToolsVisual Tools • Find images and data: Navigate • Browse objects one at a time: Explore • Find objects on a frame, compare your own observations: Finding Chart • View multiple objects or thumbnail search results: Image List SDSS catalog data demo, March 30, 2007

31. Finding the right tool (contd.) • When you don’t know specific objects… • ToolsSearch • Find objects in a patch of sky: Radial or Rectangular • Simple search for imaging data: Imaging Query Form • Simple search for spectral data: Spectro Query form • More sophisticated searches:SQL • Long, complex queries: CasJobs SDSS catalog data demo, March 30, 2007

32. Finding the right tool (contd.) • When you have a list of objects, and you want to see what SDSS knows about them… • ToolsObject Crossid • Imaging crossid: Upload • Spectro crossid: SpecList SDSS catalog data demo, March 30, 2007

33. Data access: general tips • Use collab or astro sites • Less “frills”, more direct access to tools • Some extra features • e.g. Imaging/Spectro Form Query not on public page • Each release has separate sites • http://cas.sdss.org/collab/ (the public release) • http://cas.sdss.org/collabdr6/ (not yet public) • Use Contact link to contact help-desk (not direct E-mail) SDSS catalog data demo, March 30, 2007

34. CAS Dos and Don’ts • Do not submit a query unless you have some idea how long it will take! • It could tie up the server for hours (sometimes days)! • Use “select count(*)” or “select top 10” instead • Do use predefined views for commonly-used objects and parameters • PhotoObj, PhotoPrimary for photometry • PhotoTag if you only need popular fields • SpecObj for spectra SDSS catalog data demo, March 30, 2007

35. Educational Resources • Extensive education content in SkyServer • Use Projects link on the menu at the top • K-12 and college level student exercises and teacher resources SDSS catalog data demo, March 30, 2007

36. Educational projects • Go to public site – http://cas.sdss.org • Go to Projects • Great as labs for intro classes SDSS catalog data demo, March 30, 2007

37. Focused practice • Get into groups of two • Share a computer • Consider some task you do often in research or teaching • Some ideas in your handout • Use SkyServer or CasJobs to accomplish the task • We will answer questions, offer advice • You have 45 minutes… go! SDSS catalog data demo, March 30, 2007