1 / 10

### Optimizing Data Flow and Storage for CERN's Tier 2 Network and Physics Analysis ###

This document outlines the architecture and workflow for data import, export, and storage systems in CERN's Tier 2 network. It highlights the stages of data processing from raw simulations to ESD, AOD, and DPD formats, as well as the roles of local institutes and simulation centers in managing and analyzing data. Emphasis is placed on scheduled and predictable experiments, including the analysis carried out by physics groups. The report also discusses the technical requirements for mass storage systems, database servers, and related development initiatives to optimize data accessibility and analysis capabilities. ###

stash
Télécharger la présentation

### Optimizing Data Flow and Storage for CERN's Tier 2 Network and Physics Analysis ###

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Mass Storage & Disk Servers Database Servers Data Import Data Export Tier 2 Network from CERN Local institutes Network from Tier 2 and simulation centers Production Reconstruction Raw/Sim-->ESD Scheduled, predictable experiment/ physics groups Production Analysis ESD-->AOD AOD-->DPD Scheduled Physics groups Individual Analysis AOD-->DPD and plots Chaotic Physicists CERN Tapes Tapes Desktops Physics Software Development R&D Systems and Testbeds Info servers Code servers Web Servers Telepresence Servers Training Consulting Help Desk

  2. WORKFLOW Mass Storage & Disk Servers Database Servers Data Export Data Import Tier 2 Network from CERN Local institutes Network from Tier 2 and simulation centers Production Reconstruction Raw/Sim-->ESD Scheduled, predictable experiment/ physics groups Production Analysis ESD-->AOD AOD-->DPD Scheduled Physics groups Individual Analysis AOD-->DPD and plots Chaotic Physicists CERN Experiment Tapes Tapes Desktops Physics Groups Individual Physicists Physics Software Development R&D Systems and Testbeds Info servers Code servers Web Servers Telepresence Servers Training Consulting Help Desk

  3. DATAFLOW Robotic Mass Storage Mass Storage & Disk Servers Database Servers Data Import Data Export Tier 2 Central Disk Cache Network from CERN Local institutes Network from Tier 2 and simulation centers Local Disk Cache Production Reconstruction Raw/Sim-->ESD Scheduled, predictable experiment/ physics groups Production Analysis ESD-->AOD AOD-->DPD Scheduled Physics groups Individual Analysis AOD-->DPD and plots Chaotic Physicists CERN Local Disk Cache Tapes Tapes Local Disk Cache Desktops Local Disk Cache Physics Software Development R&D Systems and Testbeds Info servers Code servers Web Servers Telepresence Servers Training Consulting Help Desk

  4. Mass Storage & Disk Servers Database Servers Data Import Data Export Data Input Rate from CERN: Raw Data - 5% 50TB/yr ESD Data - 50% 50TB/yr AOD Data - All 10TB/yr Revised ESD - 20TB/yr Data Input from Tier 2: Revised ESD and AOD - 10TB/yr Data Input from Simulation Centers: Raw Data - 100TB/yr Tier 2 Network from CERN Local institutes Network from Tier 2 and simulation centers Production Reconstruction Raw/Sim-->ESD Scheduled, predictable experiment/ physics groups Production Analysis ESD-->AOD AOD-->DPD Scheduled Physics groups Individual Analysis AOD-->DPD and plots Chaotic Physicists CERN Tapes Tapes Desktops Physics Software Development R&D Systems and Testbeds Info servers Code servers Web Servers Telepresence Servers Training Consulting Help Desk

  5. Mass Storage & Disk Servers Database Servers Data Import Data Export Tier 2 Network from CERN Total Storage: Robotic Mass Storage - 300TB Raw Data: 50TB 5*10**7 events (5% of 1 year) Raw (Simulated) Data: 100TB 10**8 events EDS (Reconstructed Data): 100TB - 10**9 events (50% of 2 years) AOD (Physics Object) Data: 20TB 2*10**9 events (100% of 2 years) Tag Data: 2TB (all) Calibration/Conditions data base: 10TB (only latest version of most data types kept here) Central Disk Cache - 100TB (per user demand) CPU Required for AMS database servers: ??*10**3 SI95 power Local institutes Network from Tier 2 and simulation centers Production Reconstruction Raw/Sim-->ESD Scheduled, predictable experiment/ physics groups Production Analysis ESD-->AOD AOD-->DPD Scheduled Physics groups Individual Analysis AOD-->DPD and plots Chaotic Physicists CERN Tapes Tapes Desktops Physics Software Development R&D Systems and Testbeds Info servers Code servers Web Servers Telepresence Servers Training Consulting Help Desk

  6. Mass Storage & Disk Servers Database Servers Data Import Data Export Tier 2 Network from CERN Data Output Rate to CERN: AOD Data - 8 TB/yr Recalculated ESD - 10 TB/yr Simulation ESD data - 10 TB/yr Data Output to Tier 2: Revised ESD and AOD - 15 TB/yr Data Output to local institutes: ESD, AOD, DPD data - 20TB/yr Local institutes Network from Tier 2 and simulation centers Production Reconstruction Raw/Sim-->ESD Scheduled, predictable experiment/ physics groups Production Analysis ESD-->AOD AOD-->DPD Scheduled Physics groups Individual Analysis AOD-->DPD and plots Chaotic Physicists CERN Tapes Tapes Desktops Physics Software Development R&D Systems and Testbeds Info servers Code servers Web Servers Telepresence Servers Training Consulting Help Desk

  7. Mass Storage & Disk Servers Database Servers Data Import Data Export Tier 2 Network from CERN Local institutes Farms of low cost commodity computers, limited I/O rate, modest local disk cache ----------------------------------------------------- Reconstruction Jobs: Reprocessing of raw data: 10**8 events/year (10%) Initial processing of simulated data: 10**8/year 1000 SI95-sec/event ==> 10**4 SI95 capacity: 100 processing nodes of 100 SI95 power Network from Tier 2 and simulation centers Production Reconstruction Raw/Sim-->ESD Scheduled, predictable experiment/ physics groups Production Analysis ESD-->AOD AOD-->DPD Scheduled Physics groups Individual Analysis AOD-->DPD and plots Chaotic Physicists CERN Tapes Tapes Desktops Physics Software Development R&D Systems and Testbeds Info servers Code servers Web Servers Telepresence Servers Training Consulting Help Desk

  8. Event Selection Jobs: 10 physics groups * 10**8 events (10%samples) * 3 times/yr based on ESD and latest AOD data 50 SI95/evt ==> 5000 SI95 power Physics Object creation Jobs: 10 physics groups * 10**7 events (1% samples) * 8 times/yr based on selected event sample ESD data 200 SI95/event ==> 5000 SI95 power Derived Physics data creation Jobs: 10 physics groups * 10**7 events * 20 times/yr based on selected AOD samples, generates “canonical” derived physics data 50 SI95/evt ==> 3000 SI95 power Total 110 nodes of 100 SI95 power Mass Storage & Disk Servers Database Servers Data Import Data Export Tier 2 Network from CERN Local institutes Network from Tier 2 and simulation centers Production Reconstruction Raw/Sim-->ESD Scheduled, predictable experiment/ physics groups Production Analysis ESD-->AOD AOD-->DPD Scheduled Physics groups Individual Analysis AOD-->DPD and plots Chaotic Physicists CERN Tapes Tapes Desktops Physics Software Development R&D Systems and Testbeds Info servers Code servers Web Servers Telepresence Servers Training Consulting Help Desk

  9. Mass Storage & Disk Servers Database Servers Data Import Data Export Tier 2 Network from CERN Local institutes Network from Tier 2 and simulation centers Production Reconstruction Raw/Sim-->ESD Scheduled, predictable experiment/ physics groups Production Analysis ESD-->AOD AOD-->DPD Scheduled Physics groups Individual Analysis AOD-->DPD and plots Chaotic Physicists CERN Derived Physics data creation Jobs: 200 physicists * 10**7 events * 20 times/yr based on selected AOD and DPD samples 20 SI95/evt ==> 30,000 SI95 power Total 300 nodes of 100 SI95 power Tapes Tapes Desktops Physics Software Development R&D Systems and Testbeds Info servers Code servers Web Servers Telepresence Servers Training Consulting Help Desk

  10. Mass Storage & Disk Servers Database Servers Data Import Data Export Tier 2 Network from CERN Local institutes Network from Tier 2 and simulation centers Production Reconstruction Raw/Sim-->ESD Scheduled, predictable experiment/ physics groups Production Analysis ESD-->AOD AOD-->DPD Scheduled Physics groups Individual Analysis AOD-->DPD and plots Chaotic Physicists CERN Tapes Tapes Desktops Physics Software Development R&D Systems and Testbeds Info servers Code servers Web Servers Telepresence Servers Training Consulting Help Desk

More Related