1 / 32

Dark Energy Survey Directors’ Review Closeout Presentation

Dark Energy Survey Directors’ Review Closeout Presentation. Fermilab May 10-11, 2011. Table of Contents 1.0 Executive Summary (none yet) 2.0 Introduction 3.0 DECam 4.0 Data Management and Community Pipeline 5.0 Installations 6.0 Calibration 7.0 Commissioning and Early Operations

raja
Télécharger la présentation

Dark Energy Survey Directors’ Review Closeout Presentation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Dark Energy Survey Directors’ ReviewCloseout Presentation Fermilab May 10-11, 2011

  2. Table of Contents • 1.0Executive Summary (none yet) • 2.0Introduction • 3.0DECam • 4.0Data Management and Community Pipeline • 5.0Installations • 6.0Calibration • 7.0Commissioning and Early Operations • 8.0Charge Questions

  3. 2.0 Introduction • A Review of the Status of the Dark Energy Survey, commissioned by the DES Council of Directors, was held on May 10-11, 2011 at the Fermi National Accelerator Laboratory. The object of this review was to assess the status and adequacy of the overall plans and preparations for Integration, Installation, Commissioning, and Early Operations of DES. The charge included a list of topics and specific questions to be addressed as part of the review. The assessment of the Review Committee is documented in the body of this closeout presentation. • Each section in this closeout presentation is generally organized by Findings, Comments and Recommendations. Findings are statements of fact that summarize noteworthy information presented during the review. The Comments are judgment statements about the facts presented during the review and are based on reviewers’ experience and expertise. The comments are to be evaluated by the DES team and actions taken as deemed appropriate. Recommendations are statements of actions that should be addressed by the DES team. The remainder of this presentation has the answers to the review charge questions. • Under its Project Director, the DES is to develop a response to the review recommendations and present it to the DES Council of Directors. A response to recommendation(s) is expected and actions taken will be reported on during future reviews.

  4. 3.0 DECam Johns, Smith, Kibrick, Cassidy, Ihle

  5. DECam Findings • The DECam team has completed or substantially completed major subsystems of the camera and has tested the assembly, without optics, on the telescope simulator at FNAL. • Polishing of the corrector lenses was completed in Jan 2011 and the lenses are being coated and installed in cells at UCL. • The DECam collimator barrel assembly has been shipped to UCL for installation of the lenses. The barrel assembly will be shipped to CTIO in two parts following installation and alignment of the optics. • The imager has been tested at FNAL populated with 56 of 62 engineering grade CCDs. Science grade CCDs are currently being installed. • The SISPI camera controller has been completed, installed at CTIO, and tested with the telescope simulator. • The F/8 secondary mirror changer has been shipped and installed at CTIO. • High impact risks in the DECam project have been retired notably those associated with the Imager and production of CCDs.

  6. DECam Findings • Asahi is 8 months behind in producing filters for the project which has placed filter delivery on the DOE watch list. Only 3 of 10 surfaces have been coated to spec, all single sided. Asahi has reportedly coated the second surface of the i-filter that nearly meets specifications. Unless a critical number of filters are in hand, DECam installation on the 4-m will be put on hold. The project is considering an alternative procurement. • The LN2 cooling system is being shipped to CTIO. Lifetime problems with the circulating pump will be dealt with in Chile. • DECambudget covers delivery and assembly of DECam “on the platform” at CTIO. The installation and commissioning of DECam on the Blanco 4-m telescope will be the responsibility of CTIO. FNAL will provide support using “operations funds.” • The DECam project has spent $31.6M to date (Mar 31). The estimated cost to completion is $34.1M out of a total budget of $35.1M. Remaining contingency is $1.0M. • The DECam estimated date of completion is in Dec 2011 with 9 months of schedule contingency.

  7. DECam Finding • The DECam Installation Review Committee meet in La Serena on April 13-14, 2011, and made specific recommendations regarding planning for pre-commissioning integration and installation. • LN2 cooling system test operated imager at -115 C, nominal is -100 C • Partial loss of full well and output damage traced to massive overexposure. Interlock developed, installed, and tested to prevent such damage. • Testing on telescope simulator is completed . • The immersed liquid nitrogen pump has failed at 2000 hours, well before its predicted life (8000 hours). Design improvements are being implemented to extend its life. Pump replacement requires a week when performed by experts, limited primarily by warming and cooling times.

  8. DECam Comments • DECam will have never been tested as an imaging optical system prior to installation on the Blanco telescope. While reasonable care has been taken in the fabrication of the lenses and installation and alignment in the barrel assembly, the project will not have confirmed that the camera meets requirements until stellar images are obtained under representative observing conditions. If the optics need to be removed from the telescope for re-work or re-alignment the cost and schedule delay could be significant. • The DECam assembly plan at CTIO is success oriented. It is the experience of many (most?) ground-based telescope projects that a significant number of problems are discovered and get corrected after installation. These can vary greatly in severity. While DECam has taken significant care in its test program at FNAL to reduce the number of these, problems will inevitably arise that require additional resources to correct.

  9. DECam Comments • The cost, plan and schedule for re-assembly and check out of DECam on the ground at the Blanco telescope is plausible. FNAL labor resources have been identified for this work. • An installation and commissioning plan, in narrative form, that includes interfaces to the mountain staff and other institutions has been developed for operations. • The installation and commissioning of major instruments at ground-based telescopes is usually the responsibility of the project team that built the instrument with assistance from the observatory operations staff. For programmatic reasons CTIO is taking the lead for DECam I&C with FNAL in a supporting role. Close coordination and planning will be required on the part of the integrated project team for this to work. An integrated plan for the installation and commissioning of DECam in the telescope is needed. At this time, the roles, chains of command, and communication interfaces between the players and institutions that will be engaged in installation and commissioning do not appear to be well defined. [move to installation]

  10. DECam Comments • CTIO will be using its operations staff augmented with one additional hire to install and commission DECam. The staff should be able to accomplish the work with close assistance from FNAL and the science teams. However the CTIO technical staff will continue to support normal telescope operations on the mountain and efficiently doing both jobs simultaneously will be challenging. • CTIO is funding its I&C efforts from its base operations budget. The FNAL DECam team has received “operations” funding to support this effort in FY2011 and has applied for additional funding in FY2012. This funding is essential. • DECam will greatly benefit from early first-look data software and from image analysis tools. • The DECam team is to be commended for finding the overexposure problem and implementing a solution.

  11. DECam Recommendations • Produce a detailed plan for assembly, installation, and commissioning of DECam at CTIO with the integrated DES project team. • Secure funding commitments to ensure adequate support during commissioning. • Hold adequate contingency funds to cover unexpected expenses during assembly and check out of DECam. • Do not purchase filters from alternate vendor unless the project determines that Asahi is incapable of completing their order. • Develop or adapt software tools in time to provide image QA and telescope alignment diagnostics early in the commissioning process. • Thoroughly characterize the imaging performance of the corrector before launching into early science programs. • Adopt recommendations of the Installation Review committee.

  12. DECam Recommendations • There might be some risk of thermal stress in CCD packages if DECam CCDs are allowed to reach temperatures significantly below -130 C. Absent a compelling reason for the imager to reach such cold temperatures, adjust imager cold straps so that with imager at CTIO the CCDs operate at the nominal -100 C operating temperature when heater resistors are operating at the center of their control range. • Replicate engineering/telemetry database at FNAL to simplify remote support • Develop contingency plans for when the success oriented installation and commissioning plan encounters unexpected problems such as optical problems with the corrector. • Provide a means for rapidly recovering from failures of the LN2 system.

  13. 4.0 Data Management Findings • The team has been very significantly restructured and redirected in the past year under the guidance of the new Project Manager, with many more resources being focused on the CP as a critical formal deliverable. • Working practices have been changed to improve task assignment, tracking and accountability. • The team has been augmented in important ways, both through targeted new hires and closer coordination with the SWGs. • SWGs have provided detailed, coordinated, quality control acceptance testing criteria for DESDM, and are using these to validate DC performance. • DC5b is currently estimated to be about 3 months behind schedule. DC6 will focus on science validation and DC7 on meeting production operations requirements. • Current funding from NSF lasts through October, 2011. An operations proposal has been submitted to NSF

  14. 4.1 Data Management Findings (continued) • A twin-track approach has been adopted, with DESDM development being tied to science goals and CP development being tied to production/performance goals. • Production operations have become a much more visible focus of the work, with the identification of critical bottlenecks (primarily data transfers associated with distributed computing and database performance), and the development of plans to address these. • With the retirement of key TeraGrid resources, NCSA is no longer the primary source of compute cycles. Proposals have been submitted for cycles at SDSC and TACC, together with requests for dedicated assistance to address porting and performance issues. • Efforts are underway to leverage work of other teams to help address issues of performance profiling (ASTA), data transfers (Globus) and data distribution (VAO). • The 12-hour CP processing requirement suggests that, with existing performance, the NOAO CP hardware requirements will be about twice what had been predicted. • Tertiary sites will be running databases other than Oracle, leading to potential issues with data migration and pipeline performance.

  15. 4.2 Data Management Comments • The Project Manager has done an exceptional job in the last year, and the team has taken a huge leap forward as a result. Issues that review panels have long been raising as concerns are finally being addressed in an appropriate manner. • Commissioning is perilously close, and there is still a large amount of work to be done with limited resources both of people and computing. • Separation of science and operations tasks into the twin-tracks has enabled significant complementary progress on both. • Some performance issues are still not fully understood, making it hard to scope the resources that will be required to address them. • The commissioning phase itself is very aggressively scheduled, with only 2.5 months between first light and the first science, especially since this will be the first time that either the CP or DESDM will see on-sky DECam data. • Commissioning of and acceptance testing requirements for the CP are still ill-defined, even though this is a formal deliverable that is a prerequisite for granting the DES telescope time.

  16. 4.3 Data management Comments (continued) • It seems highly likely that the CP will not be fully functional by the end of commissioning in April 2012. • Reliable long-term computing resources are not yet procured for either DSDEM or CP, and, at least in the short term, NCSA is no longer able to act as a failsafe for DESDM. • There is a risk of disengagement of the SWGs during commissioning • Deblending is a difficult issue and one that will be important to many science working groups. • Other outside expertise could be helpful; specific examples might be LSST on distributed data processing performance profiling and XLDB on database performance.

  17. 4.4 Data Management Recommendations • As a matter of urgency the CP commissioning plan, possibly including staged acceptance criteria to reflect the likely readiness of the CP at April 1st and September 1st, must be defined and documented at the directorate level. • Seek additional and/or extended in-kind contributions to support the continued development and deployment of CP in line with this new schedule. • Investigate the possibility of obtaining a guaranteed minimum annual allocation of appropriate TeraGrid/XD resources for the lifetime of the project. • Examine science acceptance metrics to see if they can be integrated into the QC framework to get more real-time assessment of acceptability before processing is complete.

  18. 5.0 Installation Findings • DECam calibration system (DECal) hardware arrives at CTIO before Sept. 2011 • There appears to be no requirement that the DM/CP software be ready prior to the start of DECam installation onto the telescope. • Installation is scheduled to end Dec. 2011, imager first light Jan. 2012 • DECam team has developed installation procedures for DECam during the test phase. • CTIO will be in charge of installation, FNAL DECam staff to assist on site • CTIO has hired a safety engineer to work with NOAO?s safety officer to develop a safety plan for the installation of DECam. • CTIO has identified two potential issues of two agencies working together,different safety cultures and language barrier

  19. 5.1 Installation Findings (continued) • CTIO foreman will oversee/supervise all staff involved in installation • Two sets of LN2 lines to install: - one in Coude' room for initial imager tests (pre-shutdown) - one on telescope for installed imager (post-shutdown) • Installation plan provides internal scaffolding and safety rails for staff • Cage alignment plan designed to locate/record/preserve existing optical axis • Active optics system employs both BCAMs and focus/alignment sensors in imager

  20. 5.2 Installation Comments • CTIO is responding to the recommendations noted by the DECam Installation Review Committee on April 13-14, 2011. • DECal installation risks getting lost in the shuffle of DECam installation • DECalflatfielding functionality could be tested separately using MOSAIC • There is a risk of commissioning starting before needed software tools are ready • Non-common language poses miscommunication risks between DECam installation staff • We did not see explicit safety procedures for working in high locations • Unclear if staff working in Tololo will have to leave their responsibilities in order to work on this project. • Pressurized LN2 lines pose new risks to staff (pinhole leaks -> spray into eyes) • Installation plan for removing PF cage/installing DECam cage appears thorough • Concern whether wooden floor of internal scaffolding will support 800 Kg asclaimed; need to verify this rating

  21. 5.3 Installation Comments (continued) • Sensitivity of donut analysis is unclear at present. • The existing Hartmann can be used to characterize the on- and off-axis optical performance of the corrector prior to installation of the Imager. • The DECam cage alignment plan appears sufficient • Active optics system requires the coordination of work at several external sites (SLAC, UM).

  22. 5.4 Installation Recommendations: • Consider whether DECal install and test can be moved forward in schedule • DM should be capable of single image de-trending by start of commissioning. • Need to establish clear mechanism for handoff of duties between foremen • Further work needed on training manuals for FNAL staff assisting install at CTIO • Update safety procedures (e.g., eye protection) to address risks from LN2 lines • Practice use of boomlift and internal scaffolding prior to start of cage work • Need plan to coordinate active optics software development between relevant sites • CTIO and the DECam team should identify and list all task that will require a job hazard analysis or procedure. • As part of following the DECam Installation recommendation of developing job  hazard analysis for tasks before work begins, the Project should identify and  list all  tasks that will require a job hazard analysis or procedure • Do not limit training to non-CTIO personnel only, a site specific safety  orientation identifying expected hazards should be developed for all personnel  working on the installation.

  23. 5.5 Installation Recommendations (continued) • Identify equipment that will require specialized training and limit use to  authorized personnel only. • The site specific orientation should address the potential language barriers  and how personnel should handle them. • The safety plan should list the roles and responsibilities of all personnel  on the project including the foreman that will be managing the daily work. • The foremen managing the work should be bilingual or have lead personnel  that are bilingual to assist them in communication. • Safety should be integrated into the procedures in that it is everyone's  responsibility to be safe and not rely on the oversight of the safety engineer. • All personnel should be given the authority to stop work if they think it is  unsafe or there is a deviation from a procedure or job hazard analysis.

  24. 6.0 Calibration Findings • A calibration plan exists for the main DES survey, with multiple consistency checks • Data from PreCAM plays a key role in the calibration. • The experience from the 2010-2011 PreCAM campaign on the Curtis-Schmidt was valuable, but did not yield enough data to fully populate the grid of reference star positions across the survey. • PreCAM also allowed the DECam CCDs and readout mechanisms to be tested on-sky. Some problems were found (banding in readout). The source of the problems has been identified, and is unlikely to affect regular DECam operations. • DEScal construction is nearing completion. The components are being approved for shipping. All are expected to be in CTIO by the beginning of September.

  25. 6.1 Calibration Comments: • PreCam will require additional observations to complete the reference grid—particularly in z and y where coverage has been almost exclusively limited to Stripe 82. • DES still owes a new secondary mirror to U. Michigan for the Schmidt—it should be secured before negotiating for further access to the telescope. • The Staffing levels for calibration appear reasonable—there is concern that progress on PreCam calibration not interfere with commissioning; clear division of roles will be important. • The goals of calibration appear to be achievable, but will require careful discussion between calibration scientist, survey operations, and the relevant data management code maintainers. • It is important that the project invest in verification of the calibration procedures already in the first year of operations, to make sure that no changes to procedure need to be made. This makes it imperative that the PreCam survey be finished before the end of the 2012-2013 observing campaign.

  26. 6.2 Calibration Recommendations • Plan for two PreCam campaigns—one in winter 2012 to fill in z,Y in the eastern part of the DES footprint and a longer one in 2012-2013 to finish the grid in all filters • Carefully isolate the work allocation of individuals between PreCam observing/analysis and DECam installation and commissioning. • Develop procedures to test the quality of the calibration—this will also enable commissioning of the DESDM.

  27. 7.0 Commissioning and Early Operations Findings • The commissioning plan is a rapidly evolving document, and appears to have the right steps. • The responsibility for commissioning rests with CTIO, but the plan necessitates persistent support and presence from DECam experts during the commissioning phase. • The Commissioning plan envisions ~2.5 months for commissioning, with multiple f8 blocks interspersed to represent analysis time—these are also listed as “contingency” in the schedule. • CTIO has a budget and staff plan for operations. Names associated with the roles are beginning to be finalized. • There is a plan for a “mini-Survey” to test out the procedures for early operations that is well-developed and will test the DM and operations plans

  28. 7.1 Commissioning and Early operations Comments • It is important that roles of all individuals during commissioning be defined, and that personnel from DECam be physically present. • Mechanisms for transmitting commissioning data from CTIO to Fermilab for distribution to relevant members of the DES team for additional tests with the commissioning data need to be in place. • Software and hardware sufficient to reduce DECam data (single epoch detrenting) and reveal any more subtle problems has to be available to the commissioning scientist at • There will be significant analysis needs during commissioning, and the “contingency” blocks scheduled do not account for any large problems encountered during commissioning • There is a significant risk that the personnel that needs to be actively involved in the execution of installation and commissioning will also be tasked to manage the planning/documentation of the schedule and resource allocation. CTIO needs to make sure this overcommitment doesn’t let critical tasks fall through the cracks.

  29. 7.2 Commissioning and Early Operations Recommendations: NOAO should designate the f8 blocks during commissioning as “engineering” time and/or designate shared-risk proposals to be executed during that time should the time be needed. Procedures to ensure that observers are not sent to Chile only to be told they have no time should be in place. CTIO should hire a professional trained in project management tools, to support CTIO staff by updating resource allocations, schedule, budget, and preparing reports, including whatever metrics for progress are jointly agreed by NOAO and Fermi to be necessary during the intense integration and commissioning periods

  30. 8.1 Charge Questions 1) Is the cost, plan, and schedule for integration, installation, commissioning, and early operations plausible, on-track, and sufficiently detailed? • Are the roles, chains of command, and communication interfaces between the different players and institutions that will be coming together on the mountain for installation and commissioning sensible and well understood by all the players? 2) Do CTIO and DES have sufficient budget, human resources, and infrastructure and procedures in place for efficient and safe installation, commissioning, and operations? • Evaluate the status of the DECam instrument development, with focus on the critical path items: does the DECam project have adequate resources to complete the remaining integration tasks on schedule?

  31. 8.0 Charge Questions 3) Evaluate the status of the software development: is there a realistic plan, schedule, and resources in place for the Data Management (DM) and Community Pipeline (CP) systems to be ready to efficiently process, serve, and evaluate the commissioning and early post-commissioning data? 4) Are DM and CP making steady, measureable progress toward those goals? a) Is the Operations Plan for DM credible and are resources identified to get DM to operational readiness by September, 2012 for the first full season of DES operations? b) Are the roles of the Science Working Groups in commissioning (i.e., acceptance testing) and early data analysis clearly delineated and do they have the analysis tools and computing resources in place? 5) Is there a sensible plan and strategy for early post-commissioning science operations before DECam is made available for normal community use?

  32. Thanks to: The presenters and preparers of supporting documentsthe DES teamThe directorsFermilab

More Related