Data Analysis Software Development

1. Data Analysis Software Development Hisanori Furusawa ADC, NAOJ For HSC analysis software team

2. Status Updates

3. 1. Prototype with Suprime-Cam Data • Quick analysis with in-hose software with CFITSIO library & TERAPIX software • Investigating RCM+Roobasf middleware • 1st-stage reduction, statistics measurement, mosaic/stack, catalog making • managing analysis history and data tags in database • Progress • First test in August 2008 • Basic processing of data is OK • Improving the system, especially in analysis results management • Testing the assembled system in Sep 2009

4. A Snapshot of the prototype software Statistics measurement Raw data Processed data

5. 2. Data Simulation • A field of view of HSC with artificial objects • PA, CCD arrangement, Geometrical effects (Distortion+ADC, Airmass), and Vignetting • Useful for development of a procedure of data handling • Photon tracing (turbulence in the atmosphere) led by Hamana&Miyazaki Example of FOV w/ vignetting

6. 3. Discussion of Analysis Procedure • Requirements on HSC data products • Hearing led by Strauss, Furusawa+  Specification & Design of HSC Analysis • Determining analysis procedure & Breaking down development tasks • Determining collaboration framework • System Design

7. HSC Data Analysis Pipeline (1)From DAQ to Single Frame Analysis Data Acquisition Use every frames from 1 shot. Assume CCD alignment. Raw images Standard StarAnalysis Bias DomeFlat ObjectFrame Astrometry & Distortion ExternalCatalog CreateBias WCS CreateFlat CorrectFrame StandardStars BiasFrame FluxCalib zeropoint Corrected/Mask/Noise Frame FlatFrame MeasurePSF PSF FindBrightObjects Bright object listwith PostageStamps include spatial variation FindFaintObjects PSF matching detection Faint object list Standard Catalog MeasureObjects for single frame Sky level, seeing, detection limit, … MeasureStats Frame Statistics

8. HSC Data Analysis Pipeline (2)From Mosaicking to Catalog Making HSCANA job WL WG job Feed back to single frames Filter, Area, Time range FindObjects Another Path When we process without stacking w/o stacking Astrometry with mosaic/stacked images CollectFrame FluxCalib Group of Corrected/Mask/Noise frames MeasurePSF MosaicFrame 1. Select stars and identification of same object 2. Solve optical distortion, atmospheric effect, CCD alignment, offsets between shots 3. Solve relative flux scaling FindFaintObjects MeasureObjects Standard Catalog for stacked image WG analysis tools can be added MeasureShape Mosaic Information x 5 bands StackFrame 1. Sky Subtraction 2. Match PSF 3. Warping and flux scaling MergeObjects Merged/Final Catalog For the objects NOT detected in some bands, measure corresponding parameters at the positions of objects detected in other bands. Stacked/Mask/Noise/Weight Image

9. Collaborative Framework (Tentative) • Analysis of Correcting Frames (1a) & Constructing Mosaic Images (1b) – NAOJ/IPMU • Photometric & Astrometric Calibration – NAOJ/IPMU • Science Archive – NAOJ • Detection of objects & Measurements of object parameters & PSF mapping – Princeton • Analysis Framework – KEK, UT/Physics

10. Topics for Today’s Discussion • For Weak Lensing Analysis: • Necessary Data Products (in what stage & in what format) • Necessary Extracted Parameters (Shape or profile, Size, Brightness …) & What are to be in ScienceData Archive • Roles of Analysis Software Team and WL-WG on WL analysis software

11. 1. Data Products for WL Analysis • Standard HSC-ANA will provide: • Individual Corrected Frames (CCD-by-CCD) with .. • Photometric & astrometric calibration (tolerance?) • PSF measurements across the fields (how&what?) • Mosaicing information, i.e., alignment & flux scaling • No geometrical transformation done (distortion, airmass effect remain) • Stacked Mosaic Image with Above 1 & 2. (what format?) • Standard Object Catalog (what parameters?) Any other?

12. 2. Parameters to be Extracted & Science Archive • HSC-ANA will extract standard parameters in the next slide (tentative) from standard stacked images • Standard parameters & images (standard corrected frames & stacked images) are supposed to be in Science archive Any other ideas?

13. HSC Data Analysis Pipeline - Standard Parameters to be measured Standard Parameters (errors are associated with all measurements) id Unique ID to identify object band Observed band MJD Mean time during exposure * Position * xc, yc center of objects in pixel ra, dec Equatrial coordinate gLong, gLat, eLong, eLat, … * Type * type star / galaxy indicator * Flag * flag object flag * Flux * (circle or ellipse aperture?) psfMag PSF fitted magnitude petroMag Petrosian magnitude petroRad Petrosian radius petroR50, petroR90 KronMag Kron magnitude KronRad Kron radius modelMag Model integrated magnitude extinction Galactic extinction Shape measurement related * Shape * Stokes Q and U 2nd order moment *Profile * n Sersic index reff Half light radius AB a/b axis ratio Phi Position Angle Radial Profile -> Aperture mag if needed * Size * FWHM Some dimension indicator * After MergeObjects * type -> utilize color information photoz link -> to external catalogs (NIR photometry, spectroscopic info) + WL dedicated parameters?

14. 3. WL Software & Our Roles • HSC-ANA is responsible to provide • Standard Images (Corrected frames & Stacked images) • Standard Object Parameters • Science Archive • WL-WG will determine data & parameters necessary for WL analysis and optimal methods to produce the data • WL-WG will lead Implementation of the methods, in collaboration with HSC-ANA members • The WL software can (& should) be added to the standard HSC analysis system by HSC-ANA members

15. End