1 / 5

Enhancing Data Management and Migration Processes in DIRAC Framework

This document outlines the functionalities and responsibilities of various components in the DIRAC framework, such as RequestDB, TransferAgent, and RemovalAgent. It details the processes for managing data transfers, registrations, and integrity checks, highlighting the integration of storage plug-ins and the migration strategies from DIRAC2 to DIRAC3. A timeline for testing and finalization of components is also provided, along with monitoring of key performance indicators like request latency, transfer times, and migration statuses, ensuring efficient data management.

sharon-poole
Télécharger la présentation

Enhancing Data Management and Migration Processes in DIRAC Framework

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CCRC ‘08 Readiness Andrew C. Smith

  2. DIRAC Pit -> Tier0 Responsibility • RequestDB accepts Data Management requests • Transfer agent polls RequestDB • Executes copy of file specified • Register file in AdtDB • Online Integrity Agent polls AdtDB • Polls Castor for migration status • If migrated polls for checksum value • File registered in FileCatalog • Standard interface to LFC,ProcDB,BK (changes replica flag) • Requires that the BK entry already present • Create removal request • Issued to RequestDB • Removal agent polls RequestDB • Executes removal of file Online system • Actually only marks file as candidate for removal Andrew C. Smith

  3. Status of DIRAC Components • All components tested last spring • DIRAC2 framework • Components present and heavily tested in DIRAC3 • SRM2/SRM1 and Castor/RFIO storage plug ins • Storage Element • Instantiates all storage plug ins in CS (using StorageFactory) • Can change which protocol used for upload through CS • ReplicaManager • Links StorageElement and FileCatalog functionality • RequestDB • Client and server (file and MySQL backend) present • Components to be migrated to DIRAC3 • TransferAgent (copy, copyAndRegister, replicate, replicateAndRegister) • RemovalAgent (removeFile, removeReplica, removePhysicalFile) • Online Integrity Agent • AdtDB Andrew C. Smith

  4. Timeline for testing • Simple -> Complex • Replication of dummy files to Castor (14/01) • TransferAgent to be finalised • Ensure SRM and RFIO can be used • Replication and registration in AdtDB (16/01) • AdtDB to be migrated • Replication, registration, migration status (18/01) • Online Integrity Agent to be migrated • Full Chain (21/01) • RemovalAgent to be finalised • Continual operation… Andrew C. Smith

  5. Monitoring • What are we interested in knowing? • Some possibilities…. • RequestDB • Number of requests in db, request latency • TransferAgent • Time for transfer, transfer failures (with reason) • AdtDB • Number of files awaiting migration, average time for migration • Online Integrity Agent • Files passing/failing checksum check • RemovalAgent • Files removal success/failure • File lifetime… • Service/DB maintains full lifetime of LFNs • All file status changes logged • e.g. upload to T0, migrated, checksum verified, LFC registered, BK registered, ProcDB registered, replicated to *-RAW, *-RAW checksum verified, *-RAW replica processed Andrew C. Smith

More Related