1 / 8

Advanced Data Management and Replica Optimization for High-Performance Workload Management

This document outlines the innovations in data management specifically focusing on replica management services within a high-performance computing environment. Key components include replica selection and location services, the Replica Location Service (RLS), and optimization algorithms to enhance workflow efficiency. Demonstrated with commands for data management across various testbed sites such as CERN and NIKHEF, the methodologies ensure optimal performance for large catalogs, supporting over a thousand simultaneous queries. The goal is to enhance the scalability and efficiency of workload management systems.

adler
Télécharger la présentation

Advanced Data Management and Replica Optimization for High-Performance Workload Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Middleware Demo Roadmap Workload Management (WP1) Data Management (WP2) Networking (WP7) Storage Element (WP5) Information Service (WP3) Fabric Management (WP4)

  2. Data Management Demo:Replica Selection and Replica Location Services Kurt Stockinger and Peter Kunszt WP2 – Data Management

  3. Replica Management Services VO Membership Service Replica Management Services Replica Manager Client Optimization Information Service Replica Metadata File Transfer: GridFTP … Replica Location Service RLS

  4. Testbed Sites & Replica Manager Commands edg-rm copyAndRegisterFile -l lfn:higgsCERN  LYON edg-rm listReplicas -l lfn:higgs edg-rm replicateFile -l lfn:higgs NIKHEF edg-rm listBestFile -l lfn:higgs  CERN edg-rm getAccessCost -l lfn:higgs CERN NIKHEF LYON edg-rm getBestFile -l lfn:higgs  CERN edg-rm deleteFile -l lfn:higgs LYON edg-rm listBestFile -l lfn:higgs  CERN

  5. Replica Location Service RLS • Local Catalogs hold the actual name mappings • Remote Indices redirect inquiries to LRCs actually having the file • LRCs are configured to send index updates to any number of RLIs • Indexes are Bloom Filters

  6. RLS Demo at SC2002

  7. RLS Demo Topology Today CERN lxshare0342.cern.ch Glasgow grid01.ph.gla.ac.uk California dc-n2.isi.edu Melbourne koala.unimelb.edu.au LocalReplicaCatalog CERN lxshare0344.cern.ch Glasgow grid03.ph.gla.ac.uk California dc-n4.isi.edu Melbourne wombat.unimelb.edu.au Replica Location Index

  8. SUMMARY • Replica Optimization • WP7 Network cost functions are integrated into the Replica Management functionality providing an essential functionality that was missing up to now. • This gives us the necessary framework to start work on high-level optimization algorithms. • Replica Location Service • Scalable distributed catalog as a much-needed replacement for the current Replica Catalog. • Addresses all issues brought up by the experiments. Tests have been conducted with very large catalogs • The lookup time for an entry is independent of the number of catalog. Tested for up to 108 entries. • The catalog withstands simultaneous user queries of over 1000 queries or inserts per second.

More Related