1 / 29

SRM Basic Specification for Computational Research

LBNL proposal detailing modifications and new functions for SRM Basic Specification v1.1. Includes methods, relationships, and operation details for data management in research settings.

meara
Télécharger la présentation

SRM Basic Specification for Computational Research

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. C O M P U T A T I O N A L R E S E A R C H D I V I S I O N SRM Basic SpecLBNL Proposal Arie Shoshani, Alex Sim, Junmin Gu, Kurt StockingerScientific Data Management GroupLawrence Berkeley National Laboratory

  2. Summary • LBNL proposal of SRM Basic specification based on v.1.1 • Original v.1.1 methods/arguments in BLUEProposed modifications in • Green for essential additionswith + to the v.1.1 • Skyblue for optional additions with ++ to the v.1.1 • Red for subtractions from the v.1.1 • Purple for renaming and implied updates of the v.1.1 with => • Notations:{} … tuple() … list (ordered collection)[] … set (unordered collection)_ … underline as mandatory (as opposed to optional)::= … definition operator| … choice operator (exclusive or)// … commentitalic … recommended

  3. Basically • From v.2.1.1 or in addition to v.1.1 • No space management • Automatic • minimum directory management • Ls, removeFiles • No mkdir, etc. • No permission management • Somewhat automatic • Some new functions • srmGetSystemInfo // ConfigInfo • Ping

  4. Relationship to v.1.1 and v.2.1.1 (1)

  5. Relationship to v.1.1 and v.2.1.1 (2)

  6. Relationship to v.1.1 and v.2.1.1 (3)

  7. Relationship to v.1.1 and v.2.1.1 (4)

  8. Relationship to v.1.1 and v.2.1.1 (5)

  9. Relationship to v.1.1 and v.2.1.1 (6)

  10. Relationship to v.1.1 and v.2.1.1 (7)

  11. GET • srmPrepareToGet() • In: authorizationID,arrayOfGetFileRequest,arrayOfTransferProtocols, userRequestDescription, storageSystemInfo, totalRetryTime • Out: requestToken, returnStatus, arrayOfFileStatus • In: SURLs, Protocols AuthorizationID, DesiredFileLifeTime, FileStorageTypeOut: RequestStatusSummary RequestID • Basic assumptions: UserID • Get() • In: SURLs, ProtocolsOut: RequestStatus

  12. RequestStatusSummary • RequestIDRequestTypeStatusCode // RequestErrorCodeStatusExplanation// ErrorDetailsSubmitTimeStartTimeFinishTimeEstimatedTimeToStartTotalFilesInRequestNumberOfQueuedFilesInRequestNumberOfFinishedFilesInRequestNumberOfProcessingFilesInRequest • requestToken,requestType, totalFilesInThisRequest, numOfQueuedRequests, numOfFinishedRequests, numOfProgressingRequests,isSuspended • RequestIDType (Get, Put, … )StateErrorMessage FileStatusesRetryDeltaTimeSubmitTimeStartTimeFinishTimeEstimatedTimeToStart

  13. PUT • srmPrepareToPut() • In: authorizationIDarrayOfFileRequest, arrayOfTransferProtocols, userRequestDescription, overwriteOption, storageSystemInfo, totalRetryTime • Out: requestToken, returnStatus, arrayOfPutFileStatus • In: {Dest_SURL Size }[]FileStorageType ProtocolsAuthorizationIDOverwriteModeDesiredLifeTime • Out: RequestStatusSummaryRequestID • Put() • In: Src_file_names Dest_file_names, Sizes WantPermanent Protocols • Out: RequestStatus

  14. COPY • In: { Source_SURL, Dest_SURL }[]AuthorizationID DesiredLifeTime FileStorageType OverwriteModeOut: RequestStatusSummaryRequestID • Copy() • In: Source_SURLs, Dest_SURLs • Out: RequestStatus • srmCopy() • In: authorizationID, arrayOfCopyFileRequest, userRequestDescription, overwriteOption, removeSourceFilesstorageSystemInfo, totalRetryTime • Out: requestToken, returnStatus, arrayOfCopyFileStatus

  15. MkPermanent • MkPermanent • IN: SURLs • OUT: ReturnStatus • srmChangeFileStorageType() • In: authorizationID, arrayOfPath, desiredStorageType, spaceToken • Out: returnStatus, arrayOfFileStatus

  16. PIN • Obsolete • Automatic when file is ready • In: TURLs • Out: RequestStatus

  17. UnPin • UnPin() • In: RequestID TURLs • Out: RequestStatus • UnPin => Release • In: RequestID TURLs | SURLs AuthorizationID • Out: { TURL | SURLRequestStatusCode Explanation } [] • srmReleaseFile() IN: requestToken authorizationID, siteURLs, keepSpace • Out: returnStatus, arrayOfFileStatus

  18. getRequestStatus • srmStatus(Get,Put,Copy)Request • IN: requestToken, authorizationID arrayOfFromSURLs, • Out: returnStatus, arrayOfFileSta • In: RequestID • Out: RequestStatus => RequestStatusSummary • For details: getRequestFilesStatus (previous getRequestStatus which used to include RequestFileStatus and FileMetaData) • getRequestStatus() • In: RequestID • Out: RequestStatus

  19. getFileMetaData • getFileMetaData • In: SURLs • Out: FileMetaData • getFileMetaData => LS • In: SURLs AuthorizationID IsAllLevelRecursive ListOffset ListCount infoOnly • Out: FileMetaData • srmLs() • IN: authorizationID, path, fileStorageType, fullDetailedList, allLevelRecursive, numOfLevels, offset, count • Out: MetaDataPathDetail[] returnStatus

  20. FileMetaData • SURLSizeOwnerGroupPermModeChecksumTypeChecksumValueFileStorageTypeStatusCodeStatusCodeLifetimeAssignedLifetimeLeftCreatedTimeLastModifiedTimeFileTypeOriginalSURLSubDirectoryPTR// dirEntries • SURLSizeOwnerGroupPermModeChecksumTypeChecksumValueIsPermanentIsPinnedIsCached • path, status, size, // 0 if dir ownerPermission, userPermission, groupPermission, otherPermission createdAtTime, lastModificationTime, owner, fileStorageType, fileType, // Directory or FilelifetimeAssigned, lifetimeLeft, checkSumType, checkSumValue, originalSURL, MetaDataSubPathDetail[]

  21. getProtocols • In: none • Out: list of transfer protocols • None in v.2.1 • Let’s keep this or put into the new srmSystemInfo or srmConfigInfo

  22. getEst[Get/PUT]Time • Make it Obsolete • getEstGetTime: • Out- RequestStatus In- SURLs, Protocols • getEstPutTime: • Out- RequestStatus In- Src_file_name, dest_files_names, sizes, wantPermanent, Protocols • getEstimationTime • In:Source_SURL Dest_SURL Size AuthorizationIDTransferProtocols FileStorageType • Out:RequestEstimation • None • Similar one is getStatusRequest() after request is submitted

  23. AdvisoryDelete • Obsolete • AdvisoryDelete: • Out- noneIn- SURLs • srmRemoveFiles • In:SURLs authorizationID • Out: ReturnRequestStatus • srmRemoveFiles • IN: requestToken, authorizationID, siteURLs • Out: returnStatus, arrayOfFileStatus

  24. GetRequestFilesStatus • In: RequestID ListOffset ListCount • Out: RequestFileStatus [] • In: RequestIDOut: RequestFileStatus [] • srmStatusOf{Get, Put, Copy}Request • IN: requestToken, authorizationID arrayOfFromSURLs, • Out: returnStatus, arrayOfFileStatus

  25. RequestFileStatus • Source_SURLDest_SURLSizeStatusCode Explanation TURLCheckSumType, CheckSumValueFileStorageTypeEstWaitingTimeOnQueueEstProcessingTimeSourceFilenameDestFilenameRemainingLifetime • SURLSizeStateFileldTURLOwner, Group, PermModeCheckSumType, CheckSumValueIsPinned, IsPermanentQueueOrderEstSecondsToStartSourceFilenameDestFilenameIsCached • fromSURLInfo, fileSize, • status, • estimatedWaitTimeOnQueue, • estimatedProcessingTime, • transferURL • remainingPinTime • siteURL, • toSURL,

  26. Ping • In: none • Out: alive_response

  27. GetConfig • In: flag • Out: { value | list of supported functions} • e.g. GET, PUT, COPY, DYNAMIC_MKDIR, PUSH_MODE, PULL_MODE, Default_space = 5GB/user, etc.

  28. New types • FileStorageType • VOLATILE, DURABLE, PERMANENT • FileType • FILE, DIRECTORY, LINK • OverwriteMode • NEVER, ALWAYS, WHEN_DIFFERENT • RequestType • GET, PUT, COPY • StatusCode • // RequestStatusCode, FileStatusCode, RequestErrorCode, FileErrorCode • Explanation: human-readable strings // ErrorDetails • RequestID : from int to string

  29. Conclusion • Based on SRM v.1.1 and many of v.2.1.1 • Included much of Olof’s issues and suggestions to v.1.1 • Operational specification would be bound to Web Services.

More Related