CIAO Jonathan McDowell

1. CUC Apr 2010 CXC-SDS CIAO Jonathan McDowell

2. CUC Apr 2010 CXC-SDS CIAO Update – Jonathan McDowell Community support CIAO 4.2 released late 2009; CIAO 4.3 planned for late 2010 CIAO 2010 work underway: tools, scripts, Sherpa, R&D Catalog (characterization, HRC, etc.) - covered in Ian's talk

3. CUC Apr 2010 CXC-SDS Community support • No staff changes since last CUC • Helpdesk: 150 new tickets (Oct 20-Apr 9), 10 still open • Median time to first answer 2 hours (longest 1 week, updated contrib script/staff member on vac) • Median time to final answer 15 hours, but some stay open for several months (ticket 12365, resolved with release of CIAO4.2; ticket 12358, supported undergrad who needed “handholding”) • Answers generated 17 new docs, 8 new bugs, 11 RFEs • 57% did not require scientist or DS support • 14 open tickets from previous period all now closed • Gave catalog GUI (CSCview) and CIAO demonstrations at DC AAS (Jan 2010) and Hawaii HEAD meeting (March 2010) • CIAO workshop

4. 7th CIAO Workshop CXC Quarterly Report Mar 2010 CXC-SDS

5. CUC Apr 2010 CXC-SDS CIAO 4.2 downloads CIAO 4.2 was released in December 675 downloads of 'core' CIAO4.2: Linux 399 (of which 57 were 64-bit) Mac (Intel) 191 (of which 36 were 64-bit) Mac (PPC) 16 Solaris 1 Source build 68 Still a few downloads of older versions: 5 CIAO 4.1, 4 CIAO3.4 45+ Source build downloads Conclusion: Linux and Mac-Intel remain our users' dominant platforms Mac PPC and Solaris continue to decline – very few users (Leicester is the only external Solaris download so far). PPC build is being retired.

6. CUC Apr 2010 CXC-SDS CIAO 4.2 Modular downloads CIAO 4.2 is our first 'modular' release (and the first use of the new CIAO-install method described in last quarterly). Allows us to patch parts of CIAO independently, improving our responsiveness, and provides opportunity for users to get a smaller footprint if that's all they need. You can get tools-only, sherpa-only, chips-only or a mixture Stats for a subset of 506 downloads: Full installation Only tools Sherpa/Chips Sherpa Chips Tools/S Tools/C 393 17 27 14 7 21 27 Obsvis, prism sometimes omitted from full installation; totals are incomplete and do not include source downloads Too early to draw big conclusions on the demand for parts of CIAO. We'll continue to monitor this.

7. CUC Apr 2010 CXC-SDS CIAO and Sherpa ADS citations 1999-2009 From archive group database

8. CUC Apr 2010 CXC-SDS CIAO work underway in 2010 Instruments: - Graded CTI support: testing underway - Temperature dependent CTI correction - Improvements to grating zero order analysis method - HRC-S tgain corrections and improved tgextract filters, reducing LETGS background by factor two - Improvements to contaminated bias repair in V&V PSF - Support for PSF aperture correction for ARF - Support for elliptical encircled energy files - Continuing analysis of PSF calibration and support - MARX enhanced for new contamination correction Data Model - Support for tab-separated file variant used in CSCView format - Update support for FITS WCS conventions Scripts: - Rewrites of merge-all, psextract/specextract/acisspec, reprocessing thread Docs: - Combining and merging for imaging, spectral extraction, gratings

9. CUC Apr 2010 CXC-SDS Improved scripts Examples of scripts in development: chandra_repro: A reprocessing script to automate the CIAO analysis threads. - read data from the standard data distribution primary/ secondary/ - perform data cleaning and filtering starting with level 1 data and generating new level 2 event files. - first version just for ACIS imaging data, will be extended obs_align: A script to automate the “combine” and “reproject_aspect” threads. Takes a set of event and asol files and a region containing sources; uses wavedetect to generate source lists and then matches them to adjust the WCS in each event file and aspect solution. combine_spectra: A script to combine (sum or average) imaging source spectra (with or without corresponding ARFs and RMFs) and optionally background spectra (and responses), with appropriate doc caveats that simultaneous fitting is theoretically preferable

10. CUC Apr 2010 CXC-SDS S-Lang phaseout Removing S-Lang: - slsh and S-Lang will remain in OTS for people to use in their scripts - S-Lang support removed from Sherpa and Chips and not supported - S-Lang modules (pixlib wrappers etc.) removed and not supported - Users trying to start Sherpa and Chips in S-Lang mode will be directed to a conversion web page; we will provide helpdesk support for translations - Our documentation infrastructure allows us to remove S-Lang part of the ahelp docs automatically in all but a small number of cases - Some significant work is needed to clean up the web site

11. CUC Apr 2010 CXC-SDS TGCat TGCAT (tgcat.mit.edu) now has more than 1000 observations: - ACIS HETG 729 ACIS LETG 97 HRC LETG 225 Total 1051 including 326 distinct sources We are reprocessing to include HRC-S tgain and ACIS contam updates.

12. CUC Apr 2010 CXC-SDS Special topic: Sherpa

13. CUC Apr 2010 CXC-SDS Focus on UI: new user functions to input and view the data, filter and group Updated error messages New models Improved interface to PSF convolution Focus on convergence: Improvements to optimization methods New Confidence Limit Function - conf() Calculating uncertainties on the best fit parameters is now more efficient and supersedes projection() Sherpa session can now be saved in an ASCII format with save_all() function, a number of save functions are available for specific parts of the session. Enhancements and Bug Fixes: Fully support wavelength space analysis, ARF rebinning if ARF is defined on a finer energy grid than RMF Sherpa for Python users - standalone application can be built and use outside of CIAO Sherpa 4.2 release As presented pre-release to Oct CUC:

14. CUC Apr 2010 CXC-SDS SherpaCL – a CIAO 3.4 style CLI for Sherpa and ChIPS • Available within the Sherpa contrib package ciao-4.2-contrib • Provides an interactive environment with a subset of Sherpa/Chips commands from CIAO 3.4 • Converts CIAO 3.4 commands into the equivalent CIAO 4 version where possible • Developed by Doug Burke (SDS) • Limited support and testing ciao-579: sherpacl ------------------------------------------------------- Welcome to SherpaCL: CXC's Modeling and Fitting Program ------------------------------------------------------- Version: 0.24 - April 2010 Type AHELP for help. Type SYNTAX command for the syntax of the command. Type EXIT, QUIT, or BYE to leave the program. sherpacl> sherpacl> paramprompt off Model parameter prompting is off sherpacl> convert on Convert setting is on, output to screen sherpacl> source = xsphabs[gal] * xsmekal[src] -> set_source(xsphabs.gal*xsmekal.src) sherpacl> notice energy 0.5:6 -> notice_id(1,0.5,6.0) sherpacl>

15. CUC Apr 2010 CXC-SDS Distributions of Flux and Parameters We are providing users with the ability to use simulations to get errors on derived quantities (e.g. flux) Distribution of simulated Cash stat. values should have its mode close to best fit value (equivalent of goodness-of-fit test for chi-sq) Function: sample_energy_flux http://cxc.harvard.edu/sherpa/threads/flux_dist/index.py.html Monte Carlo Simulations of parameters assuming Gaussian distributions for all the parameters Characterized by the covariance matrix, includes correlations between parameters. sherpa-19> flux100=sample_energy_flux(0.5,2.,num=100) sherpa-20> print flux100 ---------> print(flux100) [[ 2.88873592e-10 1.10331438e+00 8.40356670e-01 6.97503733e-01 2.35411369e+00 1.03580042e+00] [ 2.90279483e-10 1.10243140e+00 8.41174148e-01 7.01009661e-01 sherpa-26> plot_energy_flux(0.5,2,num=1000) * Characterize distributions: plot PDF and CDF and obtain Quatiles of 68% and 95% sherpa-30> fluxes=numpy.sort(flux1000[:,0]) sherpa-31> a95=fluxes(0.95*len(flux1000[:,0])-1) sherpa-32> a68=fluxes(0.68*len(flux1000[:,0])-1) 68% fit 95% kT FLUX FLUX Probability Distribution 68% fit fit

16. CUC Apr 2010 CXC-SDS Sherpa – new development • Complex models: • accretion disk models applied to optical-X-ray band, SED templates • Isophot fitting in 2D images • Interpolations in user models (1D interpolation is already in ciaox) • Mixing of convolved/unconvolved models • specific PHA-style spectral case • UI support for "model stacks" : • this is for "easier" definitions of models/parameters for multiple data sets in simultaneous fitting, or "deproject" models. • MCMC method prototype out this summer, full Sherpa implementation in next release • Simulations to account for calibration uncertainties • our internal code fully supports the methods, need the calibration data (ARF uncertainties) in order to release for the users.

17. CUC Apr 2010 CXC-SDS Monte-Carlo methods which probe parameter space using accept/reject criteria based on the likelihood function, and generate chains in which parameter values at next iteration depend only on results (params, Cash, etc) at previous iteration User can choose from standard 'prior' distributions: distributions of parameters (e.g. flat, normal, log-normal, bimodal etc.) Why use it? Probe parameter space and fit complex models Calculate uncertainties on parameters, flux, derived correlated or linked parameters Propagate non-symmetric errors into flux or other derived quantity Include calibration uncertainties Hypothesis tests, e.g. ppp for significance of the lines UI follows the Sherpa defaults - set_method(), fit(), get_fit_results() etc. Uses “Metropolis-Hastings” algorithm MCMC – Markov Chain Monte Carlo

18. CUC Apr 2010 CXC-SDS Fitting: Optimization Methods in Sherpa We have been carrying out detailed characterization of Sherpa's optimization capabilities. We support two main approaches to optimization: “Single - shot” routines: Simplex and Levenberg-Marquardt start from a guessed set of parameters, and then try to improve the parameters in a continuous fashion: Very Quick Depend critically on the initial parameter values Investigate a local behaviour of the statistics near the guessed parameters, and then make another guess at the best direction and distance to move to find a better minimum. Continue until all directions result in increase of the statistics or a number of steps has been reached “Scatter-shot” routines: Monte Carlo try to look at parameters over the entire permitted parameter space to see if there are better minima than near the starting guessed set of parameters.

19. CUC Apr 2010 CXC-SDS Optimization Methods: Comparison Method Number Final of Iterations Statistics ----------------------------------------- Levmar 31 1.55e5 Neldermead 1494 0.0542 Moncar 13045 0.0542 Example: Spectral Fit with 3 methods Data: high S/N simulated ACIS-S spectrum of the two temperature plasma Model: photoelectric absorption plus two MEKAL components (correlated!) Start fit from the same initial parameters Figures and Table compares the efficiency and final results Good fit Bad fit Levmar fit Nelder-Mead and Moncar fit Data and Model with initial parameters

20. CUC Apr 2010 CXC-SDS Optimization Methods: Probing Parameter Space 2D slice of Parameter Space probed by each method Statistics vs iteration Temperature vs iteration Statistics vs. Temperature levmar Local minimum neldermead minimum moncar minimum

21. CUC Apr 2010 CXC-SDS Optimization Methods: Summary • “levmar” method is fast, very sensitive to initial parameters, performs well for simple models, e.g. power law, one temperature models, but fails to converge in complex models. • “neldermead” and “moncar” are both very robust and converge to global minimum in complex model case. • “neldermead” is more efficient than “moncar”, but “moncar” probes larger part of the parameter space • “moncar” or “neldermead” should be used in complex models with correlated parameters

22. PSF Research- work in progress 2009-09-24 Joint work by M. Karovska (SDS) and M. Juda, presented at HEAD meeting AR Lac, Chandra/HRC deconvolved with simulated HRMA PSF, image made with 0.03 arcsecond pixels (images from Karovska (left), Juda (right)). Roll angles are 336, 197, 218 deg. Note extended structure at 0.8-1 arcsecond (vertical bar is 1 arcsecond long) Images were taken at three different spacecraft roll angles, and are presented in spacecraft coordinates (so it appears that the structure is fixed wrt the HRMA not the sky) Margarita Karovska and Mike Juda made independent Richardson-Lucy deconvolutions (different software, slightly different corrections and PSF models) and both see this feature. Also seen in other objects (Capella). Margarita's reconstruction done with CHART and CIAO arestore. Signal level is about 6 percent of total flux. Does not appear in pre 2002 data What's going on here? We are looking to see if we can see it in ACIS data. AR Lac, Chandra/HRC deconvolved with simulated HRMA PSF, image made with 0.03 arcsecond pixels. Note extended structure at 0.8-1 arcsecond (vertical bar is 1 arcsecond long) Images were taken at three different spacecraft roll angles, and are presented in spacecraft coordinates (so it appears that the structure is fixed wrt the HRMA not the sky) Margarita Karovska and Mike Juda made independent Richardson-Lucy deconvolutions (different software, slightly different corrections and PSF models) and both see this feature. Signal level is about 6 percent of total flux. What's going on here? Will look to see if we can see it in ACIS data. AR Lac, Chandra/HRC deconvolved with simulated HRMA PSF, image made with 0.03 arcsecond pixels. Note extended structure at 0.8-1 arcsecond (vertical bar is 1 arcsecond long) Images were taken at three different spacecraft roll angles, and are presented in spacecraft coordinates (so it appears that the structure is fixed wrt the HRMA not the sky) Margarita Karovska and Mike Juda made independent Richardson-Lucy deconvolutions (different software, slightly different corrections and PSF models) and both see this feature. Signal level is about 6 percent of total flux. What's going on here? Will look to see if we can see it in ACIS data.