CDF Experiment Fermilab Experiment Support

1. CDF ExperimentFermilab Experiment Support Stephan Lammel We, Sep26th, 2007

2. Overview • Fermilab staff from all around the Laboratory collaborate in the CDF Experiment: • Particle Physics Division, Computing Division, Accelerator Division and Technical Division • This brings a lot of expertise to the CDF Experiment • And gives the experiment access to a large variety of expert resources, • for consulting and in early stages of a project • experts to lead/support a project • testing/manufacturing places • contingency resources CDF, Fermilab Experiment Support, St.Lammel

3. Fermilab Experiment Support • Basic support as host laboratory: • installation, maintenance, technical, engineering, electronics, gas, networking, video-conferencing, safety, etc... • and beyond in especially three areas: • Upgrades/Replacements • Detector Operation • Offline Computing CDF, Fermilab Experiment Support, St.Lammel

4. Upgrades/Replacements • Data Logger Replacement • Detector Operation • Fermilab Responsibilities • Three Person Shift Crew • Remote Consumer Operator • Offline Computing • Fermilab Responsibilities • Expedited N-tuple Production • Glidein Processing Facilities CDF, Fermilab Experiment Support, St.Lammel

5. Data Logger Replacement • Original data logger for Run II was designed and built end of the 90’s for Run IIa luminosity by a team from Rochester and Tsukuba: • SGI/IRIX based, long term maintainability concern • single points of failure, redundancy not affordable • custom setup, loss of expertise concern • good but tight match to needs, little flexibility • data center buffer space, expensive to increase • complex, retro fitted to evolved data handling CDF, Fermilab Experiment Support, St.Lammel

6. Data Logger Replacement • Features of new data logger: • Use commodity hardware/solution where possible and appropriate • Share archival part of solution with D0 • Clean data handling interface, eliminate intermediate steps, collect and declare metadata directly for production use • Additional benefits of new data logger: • increased bandwidth to match DAQ/trigger bandwidth • 100 MB/s (from max 24 MB/s of original data logger) • Additional flexibility for high luminosity trigger environment • Increased buffer space and ability to archive data faster than to record them CDF, Fermilab Experiment Support, St.Lammel

7. Data Logger Replacement CDF, Fermilab Experiment Support, St.Lammel

8. Data Logger Replacement • Data logger replacement was a joint effort of Tsukuba, Rochester, Fermilab-PPD and Fermilab-CD • Fermilab-PPD brought in DAQ expertise • Fermilab-CD brought in metadata management and tape archival expertise • Tsukuba and Rochester brought in the expertise of the old data logger and did a lot of the integration work • under the leadership of Fermilab-PPD CDF, Fermilab Experiment Support, St.Lammel

9. Upgrades/Replacements • Data Logger Replacement • Detector Operation • Fermilab Responsibilities • Three Person Shift Crew • Remote Consumer Operator • Offline Computing • Fermilab Responsibilities • Expedited N-tuple Production • Glidein Processing Facilities CDF, Fermilab Experiment Support, St.Lammel

10. Detector Operation • Fermilab groups have agreed responsibilities for: • solenoid (sole responsibility) • silicon detector • COT (sole responsibility) • hadron calorimeter • muon systems • trigger (level-2 Pulsar, muon) • DAQ (sole responsibility) • consumer/monitoring CDF, Fermilab Experiment Support, St.Lammel

11. Detector Operation • Last year the operations department reviewed data taking, how to streamline operations, gain efficiency, etc. • The control room shifts were re-organized from four people to three of which one can be remote: • automate more common tasks to make the job easier and allow the shift to focus on critical decisions • transfer responsibilities to achieve more natural grouping and to better balance work load CDF, Fermilab Experiment Support, St.Lammel

12. Detector Operation • Three person shift crews started Dec 29th after several months of planning, review and preparation: • training of shift crew • computer console re-arrangement • automation of routine monitoring tasks • review of alarms and make selected ones audible • enhance interface to Tevatron • automate DAQ recovery for more frequent inhibits • two week trial run • Total savings of 4 FTE (3 shifts, 7 days/week) ! • Shift crew reduction was initiated and organized by Fermilab scientists CDF, Fermilab Experiment Support, St.Lammel

13. Detector Operation • Consumer operator, CO, monitors the data taking and quality. • He interacts with the scientific coordinator and detector/data taking expert but does not himself control the detector or data taking • With many collaborators in Europe and Asia participating in the detector operation from off-site is attractive to both the experiment and foreign institutions. CDF, Fermilab Experiment Support, St.Lammel

14. Detector Operation • Steps to realize a remote consumer operator: • The CO setup of the control room was duplicated in Pisa • All monitoring plots made web based by the consumer/monitoring team • IP based video link setup • A week long trial run with a remote CO and shadow CO in the control room in November • Since January one week-long remote owl shift per month from Pisa • Remote CO shifts from Tsukuba after the shutdown • Remote CO was initiated by a Fermilab scientists and setup together with a colleague in Pisa. CDF, Fermilab Experiment Support, St.Lammel

15. Detector Operation • Both the current operations heads are from Fermilab. • Three of the four associate heads are from Fermilab. • Half of the detector/online groups are lead by Fermilab people. CDF, Fermilab Experiment Support, St.Lammel

16. Upgrades/Replacements • Data Logger Replacement • Detector Operation • Fermilab Responsibilities • Three Person Shift Crew • Remote Consumer Operator • Offline Computing • Fermilab Responsibilities • Expedited N-tuple Production • Glidein Processing Facilities CDF, Fermilab Experiment Support, St.Lammel

17. Offline Computing • Fermilab group has agreed responsibility for: • data handling, including metadata administration, tape archival and disk caches • full event reconstruction • central processing facilities • analysis framework and infrastructure • databases (software, interfaces, management) • offline operations • software release management/support • various reconstruction/analysis packages CDF, Fermilab Experiment Support, St.Lammel

18. Offline Computing • The CDF analysis strategy is an evolution from earlier hadron collider experiments: • validate/tune reconstruction on small data subset • central store of reconstructed data • analysis through iterative selections • Although offline computing is based on commodity hardware/software, resources are nevertheless tight. • avoid duplication as much as possible • shared solutions to share support/maintenance • short latency/fast turn-around CDF, Fermilab Experiment Support, St.Lammel

19. Offline Computing • CDF Computing Model (data flow view): • Raw data are written to tape (base of all physics) • Detector/data are calibrated (4-6 weeks effort) • Full reconstruction using on-site production facilities • Make highly-reduced analysis datasets, N-tuples • Monte Carlo generation/simulation/reconstruction • with run config matching data taking conditions • N-tuple selection for individual analyses needs • Further analysis on user desktop and at home institutions CDF, Fermilab Experiment Support, St.Lammel

20. Offline Computing • CDF Computing Model (hardware view) • Enstore system provides IP-access tape storage • On-site processing facilities are used for: • detector/data calibration • full reconstruction • N-tuple making • analysis data selections (of the N-tuples) • Off-site processing facilities (CDF specific and Grid) • Monte Carlo generation/simulation/reconstruction • Disk cache of tape, disk pool, and project space hold the data analysis accessible CDF, Fermilab Experiment Support, St.Lammel

21. Offline Computing • Data are grouped into few months periods by the offline to handle detector re-calibration needs: • datasets for calibration produced within ½ week • calibration plus validation within 3 weeks • full reconstruction finishes a month later CDF, Fermilab Experiment Support, St.Lammel

22. Offline Computing • N-tuple (highly reduced analysis dataset) making issues and problems: • making N-tuples is very I/O intensive • it is non-trivial (need to learn N-tuple creation tricks) • physicists need to use the heavy analysis framework • making N-tuples takes quite a bit of CPU (latest analysis-level corrections are applied) • significant load on the central processing facilities • significant effort per analysis for each physicists CDF, Fermilab Experiment Support, St.Lammel

23. Offline Computing • New N-tuple production: • common framework produces the 3 most used formats • shared expertise and resources • reduced load on on-site processing facility (N3) • fast, 3 day turn-around • off-loads N-tuple making from analyses • Expedited N-tuple production great success: • individual analyses start from much smaller production N-tuples • analyses are expedited • Fermilab group developed and manages expedited N-tuple production (from the need to reduce load on central processing facilities) CDF, Fermilab Experiment Support, St.Lammel

24. Offline Computing • Currently three on-site processing facilities: • production farm (does full event reconstruction) • “CAF” (executes user analysis jobs, CDF-only) • “FermiGrid” (latest user facility, unused cycles are being shared with other Fermilab experiments) • Working on evolving processing facilities into Grid based facilities with one on-site and one off-site job submission point for users: • idea and early development of glide-in mechanism by Fermilab-CDF person • development into common Grid tool by Fermilab-CMS CDF, Fermilab Experiment Support, St.Lammel

25. Example, Higgs Search • The search for the Higgs is the top priority for the CDF Experiment. • Additional sensitivity, beyond the increased integrated luminosity, needs to come from: • increased lepton acceptance • forward b-tagging • improved di-jet mass resolution CDF, Fermilab Experiment Support, St.Lammel

26. Example, Higgs Search • A Fermilab scientist is leading the effort to trigger on muons that fall into the phi gaps between wedges. • Forward, BMU muons have been pioneered by an ex-University of Wisconsin Fermilab scientist. • Fermilab postdoc modified tracking chamber digitization for increased hit efficiency at high luminosity. • A Fermilab scientist has reworked the tracking software for increased acceptance in the forward region and to get good efficiency at high luminosity: • forward b-tagging • lepton acceptance • A group to work and improve the di-jet mass resolution was setup by CDF last autumn. It is co-lead by a scientist of Fermilab. CDF, Fermilab Experiment Support, St.Lammel

27. Sumup • Fermilab contributes with 25 FTEs (of 65 total) to the detector operation. • Fermilab contributes with 12 FTEs (of 25 total) to the offline computing. (three of the five top leaders are Fermilab) • Of the top experiment leaders/managers 5 (of 7) are Fermilab scientists. CDF, Fermilab Experiment Support, St.Lammel

28. Summary • Fermilab continues to support the CDF experiment way beyond its host laboratory duties. • With upgrades winding down Fermilab scientist are looking for ways to increase the efficiency, shorten analysis turn-around, and reduce manpower. • Strong Fermilab participation in all main support areas of the experiment including collaboration leadership. CDF, Fermilab Experiment Support, St.Lammel