1 / 9

Use of SRM File Streaming by Gateway

Use of SRM File Streaming by Gateway. Alex Sim Arie Shoshani May 2008. File Streaming – how it works. large request for files. large request for files. Get file. Release file. Small file request. user quota. user quota. Get file. user quota. user quota. Release file.

lei
Télécharger la présentation

Use of SRM File Streaming by Gateway

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Use of SRM File Streaming by Gateway Alex Sim Arie Shoshani May 2008

  2. File Streaming – how it works large request for files large request for files Get file Release file Small file request user quota user quota Get file user quota user quota Release file

  3. File Streaming – what’s the advantages • Can accommodate very large requests with a limited quota • No waste of space: use only space needed for files • Reuse space as soon as files are transferred and “released” • Share files that multiple users ask for • Keep “popular” (hot) files in cache as long as space permits • Can have smaller quotas to accommodate more users • Length of lifetime can be longer, as long as files are released(can avoid cutting off request before they finish) • With DML, transfer can be started right away, even if only someof the files are in cache • Overlap transfer to cache (from archive or another site) with transfer to User

  4. Scenario 1: Simple Scenario for User File Access • Simple http or wget download from ESG Gateway site • User goes to Gateway, selects files, requests files • User’s quota must be sufficient to hold all requested files (otherwise request refused) • Gateway gets files into SRM disk from local or remote sites • User finds out the status from the Gateway either by browser or email • User downloads files either by clicking file links on the browser or by wget • User “clicks” Gateway to release files that are downloaded (no way to enforce that) ORNL LBNL/NERSC User’s machine NCAR ESG Gateway User’s browser SRM SRM Browser Or wget SRM http transfer Disk Cache Disk Cache Disk Cache Disk Cache ORNL/HPSS NERSC/HPSS NCAR/MSS

  5. Scenario 2: Download with DML from ESG - No File Streaming • User downloads DataMoverLite (DML) • User goes to Gateway, select files, requests files • User’s quota must be sufficient to hold all requested files (otherwise request refused) • Gateway submits the request to the SRM. • SRM returns the request token to the user. • User launches DML with request token. • DML checks the status of the request and starts transferring files from Gateway to the user’s local disk(right away for files already in cache) • DML downloads files and sends “release” to SRM through the Gateway, so space is not wasted • After file transfers are completed, DML sends “request completed” to Gateway) ORNL LBNL/NERSC User’s machine NCAR request ESG Gateway User’s browser request token release DataMover Lite SRM SRM SRM Disk Cache Disk Cache HTTP/HTTPS transfer Disk Cache Disk Cache ORNL/HPSS NERSC/HPSS MSS

  6. Scenario 3: Download with DML from ESG - File Streaming • File Streaming • When files fit user quota space: DML starts downloading files (right away for file already in the cache) • When files do not fit the user quota space, DML uses the “file streaming” feature (DML repeatedly sends Status to Gateway, and downloads available files ) • SRM brings files into Gateway’s SRM cache only as many files as it can fit into the user’s space quota. • As DML downloads files and sends “release” to SRM through the gateway, more files are being brought in, and “streamed” to the client. ORNL LBNL/NERSC User’s machine NCAR request ESG Gateway User’s browser request token Status release DataMover Lite SRM SRM SRM File transfer Disk Cache Disk Cache Disk Cache Disk Cache ORNL/HPSS NERSC/HPSS MSS

  7. File Streaming User’s machine Browser DataMover Lite request token User’s browser Disk Cache request token File Selection and request release Status File transfer NCAR Data Nodes LBNL/NERSC ORNL ESG Gateway SRM SRM SRM GridFTP FTP HTTP SRM request request token Disk Cache Disk Cache Disk Cache Disk Cache ORNL/HPSS NERSC/HPSS MSS

  8. Scenario 4: Download with DML from any data source locations • User downloads DataMoverLite • User goes to Gateway, select files • Gateway gets files information with source location (Gateway does not request any files) • User launches DML with files info provided by the Gateway • DML contacts Data Source with appropriate transfer protocol(security mechanism needs to be worked out for GSI access) • When a file is available, DML downloads to its local disk • DML releases files when done transferring. • DML reports statistics to the Gateway User’s machine Gateway SRMs with MSS (1) Find file(s), Get SURLs ESG Gateway User’s browser SRMs with Disk (4) Release file(s) (2) Request file(s) From NCAR’s MSS HTTP DataMover Lite (2) Request file(s) SRM (4) Release file(s) GridFTP (3) transfer file(s) Disk Cache Disk Cache (3) transfer file(s) FTP NCAR’s MSS

  9. Status of BeStMan Java Interfaces • Provide API to find out cache usage • Done: srmPing provides Total_space and Used_space • Provide API for request status summary • Done: srmRequestSummary provides no_of_files(requested, in-cache, released, in-queue) • Provide API to get all available Transfer URLs within a user’s quota • Done: srmRequestStatus provides that. • Provide API to abort a request • Done: srmAbortRequest aborts request and releases all files • Gateway needs to estimate total size of request • in order to check if request fits in quota • Accordingly, it advise user what to use • Discuss: should wget be used in streaming mode (or al-in-cache mode)?

More Related