1 / 25

An Intelligent Rule-Oriented Data Management System

This presentation outlines the intelligent rule-oriented data management system developed at the San Diego Supercomputer Center (SDSC) to facilitate data handling and accessibility. It discusses the architecture and infrastructure of the Storage Resource Broker (SRB) and how it integrates distributed resources using uniform interfaces. Key projects and future plans are highlighted, including collaborations across various scientific domains to enhance data management and user experience through efficient metadata handling and attribute-based access. Explore innovative solutions developed since 1997.

vlad
Télécharger la présentation

An Intelligent Rule-Oriented Data Management System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DataGrid An Intelligent Rule-Oriented Data Management System Wayne Schroeder San Diego Supercomputer Center, University of California San Diego

  2. Talk Outline • Background • Brief Overview of the SDSC SRB • Current Projects/Usage • Activities/Plans • Rule-Oriented Data Management System • iRODS Requirements/Planning • Architecture • Infrastructure Development • Collaborations/Plans

  3. Data delivered Ask for data • The data is found and returned • Where & how details are hidden Using a Data Grid – in Abstract Data Grid • User asks for data from the data grid

  4. DB Storage Resource Broker Metadata Catalog Storage Resource Broker Using a Data Grid - Details • User asks for data • Data request goes to SRB Server • Server looks up data in catalog • Catalog tells which SRB server has data • 1st server asks 2nd for data • The data is found and returned

  5. Using a Data Grid - Details DB MCAT SRB SRB SRB SRB SRB SRB • Data Grid has arbitrary number of servers • Complexity is hidden from users

  6. Storage Resource Broker A Data Grid Solution • Collaborative client-server system that federates distributed heterogeneous resources using uniform interfaces and metadata • Provides a simple tool to integrate data and metadata handling – attribute-based access • Blends browsing and searching • Developed at SDSC • Operational for 7+ years; • Under continual development since 1997; • Customer-driven

  7. Some SRB Features The SRB is an integrated solution which includes: • a logical namespace, • interfaces to a wide variety of storage systems, • high performance data movement (including parallel I/O), • fault-tolerance and fail-over, • WAN-aware performance enhancements (bulk operations), • storage-system-aware performance enhancements ('containers' to aggregate files), • metadata ingestion and queries (a MetaData Catalog (MCAT)), • user accounts, groups, access control, audit trails, GUI administration tool • data management features, replication • user tools (including a Windows GUI tool (inQ), a set of SRB Unix commands, and Web (mySRB)), and APIs (including C, C++, Java, and Python). SRB Scales Well (many millions of files, terabytes) Supports Multiple Administrative Domains / MCATs (srbZones) And includes SDSC Matrix: SRB-based data grid workflow management system to create, access and manage workflow process pipelines.

  8. SRB Projects • Astronomy • National Virtual Observatory • Data Grids • UK e-Science CCLRC • Teragrid • Digital Libraries and Archives • National Archives and Records Administration • National Science Digital Library • Persistent Archive Testbed • Ecological, Environmental, Oceanographic • ROADnet • Southern California Earthquake Center • SIO Digital Libraries • Molecular Sciences • Synchrotron Data Repository • Alliance for Cellular Signaling • Neuro Sciences • Biomedical Information Research Network • Physics and Chemistry • BaBar • Many others Over 650 Tera Bytes in 106 million files

  9. SRB Scalability • Over 2 Petabytes World-wide • Major SRB instances in the UK, Australia, Taiwan, US • United Kingdom - UK e-Science • Australia - APAC • Taiwan - Academia Sinica, NCHC • Europe -IN2P3, Italy, Norway • United States • 660 Terabytes at SDSC • 100 Million files • SAM QFS, HPSS, Unix file system, SRB Bricks

  10. SDSC Hosted SRB Data

  11. Case Study: SRB in BIRN BIRN Toolkit Collaboration Applications Queries/Results Data Management Viewing/Visualization Mediator GridPort Grid Management Data Model Database Scheduler Database Data Grid Computational Grid NMI SRB Globus MCAT Data Access HPSS File System Distributed Resources

  12. Federated SRB Operation Peer-to-peer Brokering Read Application in Boston Parallel Data Access Logical Name Or Attribute Condition 1 6 5/6 SRB server 3 SRB server 4 SRB agent 5 SRB agent Durham 2 San Diego Server(s) Spawning R2 R1 MCAT 1.Logical-to-Physical mapping 2. Identification of Replicas 3.Access & Audit Control Data Access R2

  13. Application Resource, User Java, NT Browsers Prolog Predicate C, C++, Linux I/O Unix Shell Third-party copy Web User Defined SRB Remote Proxies MCAT Databases DB2, Oracle, Sybase Archives HPSS, ADSM, UniTree, DMF File Systems Unix, NT, Mac OSX HRM Dublin Core DataCutter Application Meta-data SDSC Storage Resource Broker & Meta-data Catalog

  14. IRODS - the Next Generation of Data Grid Technology

  15. MCAT1 Server1.1 Server1.2 MCAT3 Server3.1 MCAT2 Server2.2 Server2.1 Moving Forward, a Two-Prong Plan • Maintain and Adapt SRB to New Usages: • SRB has reached a Stable Plateau • Bug Fixes • Some New Features • Merge Features Developed by others • Continue Testing • Improve Documentation • Continue Application Support • Existing and new Projects • Continue Answering User Queries • Chart New Areas • Federation Research - ZoneSRB • Collaborative Data Grids • Real-time Data Grids - • Virtual Object Ring Buffer • Sensors and Video Streams • Collaborating Observatories • SRB Workflows - New UI for Admins and users • Kepler actors, Matrix, etc • iRODS - Adaptive Middleware Architecture

  16. Continuing SRB Support • 10 FTEs SRB • 5 FTEs iRODS • iRODS Developers Support SRB

  17. Next generation Data Architecture • SRB is quite complex – with too many functions and operations • The intelligence is hard-coded • extensions/modifications require extreme care • But, the modules are fairly robust and reusable • AIM: Can we make SRB more flexible • Easy to customize at finer level • Example: Higher authentication for a particular collection • Example: Can we use stricter authorization for a collection • Example: Can we treat a particular resource differently • Currently- needs code changes • Solution: Use rule-based architecture to provide flexibility

  18. iRODS • A New Paradigm in Middleware Development • Flexible Collection management • Can be customized at user/collection-levels, … • Language for Collection management • As in stored procedures, triggers (RDB) • Administrative ease • Lot of potential beyond SRB • adaptive middleware architectures • This will be a fully Open Source effort

  19. Client Interface Admin Interface Rule Invoker Service Manager Rule Modifier Module Config Modifier Module Metadata Modifier Module Resource-based Services Rule Consistency Check Module Consistency Check Module Consistency Check Module Micro Service Modules Engine Current State Confs Metadata-based Services Rule Base Meta Data Base Micro Service Modules Rule-Oriented Data Systems Framework Resources

  20. Client Operation such as srbObjCreate Server-side Client-side Rule Checking Condition checking, rule firing Setup state and interact with RCAT – updates and modifications to persistent state Establish State Data Movement Backend Processing Micro Services Cleanup state and interact with RCAT – updates and modifications to persistent state CleanUp Rule-oriented Data System (Phase I Operational Model)

  21. Rules and Constraints • Rule-based • Lower-level Functions are composed of micro-services • Higher-level Functions are composed of rules of lower-level micro-services • Rules are interpreted using a rule engine • Customizability • Problems with rule composition • Integrity checks to make sure rules do not break higher-level functionalities • Declarative programming • Rules define semantics • Operational programming • Rule invocation provides procedural interpretation • Rules can be used as “checks and balances” to make sure that collections are self-consistent • Example: Rule makes two copies of each files • Constraint checking: can be used to see if the collection is consistent with this rule

  22. Rule Scalability and Decidability • Distinct Sets of Rules Applied in Different Ways • Atomic • Deferred (state flags) • Compound • Applied Using Micro-services • Granularity • User Input to Influence Rule Expression • Administration Enforcement • Collection Consistency Management • Rule Properties • Metadata Managing Execution (granularity, periodicity) • Metadata Defining Result of Rule Execution

  23. Sample Rules chkCond1(S) :- user(S) == ‘adil@cclrc’. chkCond1(S) :- coll(S) like ‘*/scec.sdsc/img/*’. chkCond2(S) :- user(S) == ‘*@nara’. chkCond3(S) :- user(S) == ‘@salk’. chkCond4(S) :- user(S) == ‘@birn’ , datatype(S) == ‘DICOM’. [OprList] implies delay for later or send to a CronJobManager Opr||Opr implies do them in parallel Opr, Opr implies do them serially • ingestInCollection(S) :- /* store & backup */ • chkCond1(S), ingest(S), register(S) • findBackUpRsrc(S.Coll, R), replicate(S,R). • ingestInCollection(S) :- /*store & check */ • chkCond2(S),computeClntChkSum(S,C1), • ingest(S), register(S), • computeSerChkSum(S,C2), • checkAndRegisterChkSum(C1,C2,S). • ingestInCollection(S) :- /* store, chk, backup & chk */ • chkCond3(S),computeClntChkSum(S,C1), • ingest(S), register(S), • computeSerChkSum(S,C2), • checkAndRegisterChkSum(C1,C2,S), • findBackUpRsrc(S.Coll, R), replicate(S,R) • computeSerChkSum(S,C3), checkAndRegisterChkSum(C2,C3,S). • ingestInCollection(S) :- /*store,check, backup & extract metadata */ • chkCond4(S),computeClntChkSum(S,C1), • ingest(S), register(S), • computeSerChkSum(S,C2), • checkAndRegisterChkSum(C1,C2,S), • findBackUpRsrc(S.Coll, R), [replicate(S,R) || extractRegisterMetadata(S)]. • ingestInCollection(S) :- /* just store */ ingest(S), register(S).

  24. New DataGrid Technology • Next Generation SRB -- iRODS: Intelligent Rule-Oriented Data Systems • Customizable and Flexible – User Configurable • Administratively Simpler – Admin Configurable • Build upon the experience of SRB Data Grid • Transition from SRB to iRODS • Client-level similarity • Meta Catalog transition • Current NSF Funding • Information Technology Research • 2 years • ~ 2 FTEs • Simple proto-type in a year • Started September 2004 • Rule-based architecture • Follow-on funding • NARA • NSF

  25. iRODS Collaborations • SRB/iRODS Developers • Arcot Rajasekar • Michael Wan • Wayne Schroeder • Other SRB Team Members • Collaborative Development • UK e-Science • University of Queensland • University of Maryland • Others

More Related