1 / 5

Efficient Cache Management and File Transfers in Data Systems

This guide outlines the process for effective cache management and file transfers within a data management system. It covers the communication between cache manager (CM) and queue manager (QM), setting parameters like FreeDisk, and steps for staging and purging files based on size limitations. The interaction with the AMS interface is also discussed for mapping file paths, maintaining data consistency, and addressing timeout mechanisms. Additionally, it highlights obtaining file names and sizes using HPSS Client API for seamless operations while managing disk space efficiently.

prue
Télécharger la présentation

Efficient Cache Management and File Transfers in Data Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Cache Management and File Transfers • Start cmanager • read cm.config and set FreeDisk parameter • Receive request from QM to stage/purge file (FID) • Stage: get file name and file size • if size less than FreeDisk • set FreeDisk = FreeDisk - size • start transfer and inform QM • else • inform QM that disk is full • Purge: • delete file • set FreeDisk = FreeDisk + size

  2. Obtaining File Names and Sizes • cmanager gets FID from QM • from FID gets name from Objectivity • from name gets size from filesize.txt • filesize.txt (ascii) lists names and sizes • if it gets too large need to use FIDs and sizes (binary) • with HPSS Client API can get sizes at run time • either through hpss_Fstat or (AMS) oofs_getsize

  3. CM - AMS Interface • AMS keeps a mapping between logical file pathnames (as known by Objectivity) and physical ones (either HPSS or local disk) • CM will talk to AMS before staging and after purging files • check or change mapping • Coordination between CM and AMS is still an issue • AMS uses a timeout mechanism • CM has more information and wants to make use of it

  4. cm.config

  5. filesize.txt

More Related