Start-Up of FEST

1. FEST Start-Up - Niko Neufeld Start-Up of FEST Niko Neufeld, CERN-PH Tuesday, Jan. 27th 2009

2. Acknowledgements FEST Start-Up - Niko Neufeld • FEST is an endeavor of the whole collaboration, in particular, the online, offline and physics (HLT) teams, so better not to thank anybody in particular: nobody mentioned == nobody forgotten 

3. What is the Full Experiment System Test FEST Start-Up - Niko Neufeld • Run simulated data through the whole Online system and the whole Offline chain because: • we had no data in 2008... • we should be ready to process data • hope to get several kHz with the first collisions • 1 hour is equivalent to ~ 7 million events… • Where do we start? • Inject events at into the High Level Trigger (the farm), as if they were coming from TEL/UKL1s • Run the HLT, to exercise control/configuration, trigger configuration, propagation of conditions and alignment constants, express stream, etc… • Several FEST sessions: 26/01 – 30/01; 02/03 –06/03

4. Replacing a 68 Million detector FEST Start-Up - Niko Neufeld …by a lot of simulated events and one powerful computer

5. Design Principles FEST Start-Up - Niko Neufeld • FEST should not only exercise the Online and Offline systems, the trigger, the Monitoring, Quality Checking and the prompt (Online) reconstruction • FEST should help answer operational questions: e.g. “What is the best way to update alignment / calibration constants?” • FEST should be operationally as close as possible to the “real thing” (like Pepsi Cola not like Virgin): • Same run-control, use the TFC to set trigger rate, self-regulate rate • This allows re-using FEST infrastructure later for dry-run tests of various components of the system

6. (Extra) Input FEST Start-Up - Niko Neufeld • New software for DAQ and control of injector • Some adaptations (network, TFC) • 100 Million min. bias events (@ 5 TeV, neg. polarity, VeLo closed) ~ 240 input files

7. The FEST menu of the day FEST Start-Up - Niko Neufeld • Commission the injector • Pass data through the HLT on a sufficient number of servers (8 – 12) • Start with a low rate, periodic trigger of ~ 100 Hz • Check the event building and writing of data to local storage • Establish the maximum trigger rate at which the injector works (self-regulating!) • Fix all problems, but don’t spend time to increase performance above 100 Hz • Do not yet send to the Offline • Commission the main HLT feature: Express line, plus other • Get the express stream working: second stream in storage, events properly routed • Verify rejection on the normal stream? • Verify the output data. They should contain an HLT result bank GALA WEDDING MENU (Minimum of 75 people) RECEPTION Host Beverage Package Includes 5 Hours Open Bar Service HOT & COLD HORS D'OEVURE TRAYED AND SERVED Hearts of Palm Strudel with Remoulade Sauce Ragout of Marinated Grilled Vegetables on Toasted Baguette Mini Beef Filets with Horseradish Cream on Crostini Tamarind Glazed Lamb Kebobs Cumin Coconut Skewered Chicken Ginger Tuna on Rice Cracker with Wasabi Aioli DINNER Domestic House Wine with Dinner (1 bottle of red wine, 1 bottle of white wine per table) P.E.I. Gold & Sweet Potato Soup with Smoked Bacon and Coriander Cress Antipasto Platter with Grilled Peppers, Spiced Olives Artichokes, Air Dried Beef and Asagio Cheese Agnolotti with Quattro Formaggio Roasted Garlic and Basil Infused Cream sauce Individual Filet of Beef with Assorted Baby Vegetables Rosemary and Pancetta Potato Rosti Black Truffle Sauce (All meals served with Assorted Rolls and Butter) Baby Greens with Sundried Tomato and Oregano Vinaigrette Grilled Shrimp and Scallop Brochette with Citrus Butter Sauce Bailey's Parfait with Burnt Almond Brittle Bourbon Chocolate Sauce Coffee or Tea Midnight Coffee Station From the Niagara Falls Hilton

8. FEST tests 2) FEST Start-Up - Niko Neufeld • Online Monitoring • Monitoring of the HLT: Counters, rates, rejectionMonitoring of sub-detectors with ‘normal’ eventsAre plots / histograms filled?Are they useful to identify SD problems?This requires SD experts around -> Advertise at the Commissioning meeting • Reconstruction farm (“prompt reconstruction”) • Sample events and run BrunelMonitor the performance: Rate, data flow.Monitor output, histograms? • Send to Offline • Start with the same low rate: A file every 10 minutes at 100 HzVerify file-transfers, update of LFC entries and Bookkeeping DatabaseIdentify (additional) tools needed to get good control of the systemHandle the Express line.

9. Fest tests 3) FEST Start-Up - Niko Neufeld • Data quality checks • How to give the green light for processing? Tools? • Processing of the express line, alignment and calibration checks • Verify DQ procedures. How do they feed back? • Run Brunel + streaming (Offline) • Setup a workflows • Distribute the processing of the files, setup the streaming • Test first at low rate, with runs of a few files... • Test at higher rate (last but not least) • Once things are OK at low rate, try to run at 1 kHz for some time

10. Some FEST facts FEST Start-Up - Niko Neufeld • Software versions: • Online v4r19, Moore v6r0, Brunel v34r1 • Events: • 100 Million min. Bias @ 5 TeV, neg polarity

11. FEST operation FEST Start-Up - Niko Neufeld • Like a global commissioning week: • Online & Offline teams present (physically or remotely) • Subdetector experts for Monitoring and Data Quality • Control-Room packed • Unlike a GCW: FEST interferes only mildly with sub-detector work: other partitions can run in parallel

12. First FEST results FEST Start-Up - Niko Neufeld Screenshot from Presenter with Calo data (4 million min bias)

13. Conclusion FEST Start-Up - Niko Neufeld • FEST is the last in a time-honored tradition of data-flow & processing exercises: RTTC (mostly Online), CCRC (mostly Offline), FEST (Online, Offline & HLT) • By far the most ambitions & most useful of these • It is very close to the “real thing”:  it needs a good amount of effort and work, but it will make us ready for the “real thing” in July! • FEST’s biggest success will be if after FEST we never have to go back to it!