L-Store Distributed storage system

1. L-Store Distributed storage system Alan Tackett Vanderbilt University Joint Project Vanderbilt - ACCRE (namespace and glue) LoCI - UTK - Micah Beck, Terry Moore (storage protocol) Nevoa Networks - Hunter Hagewood (end user tools)

2. L-Store Goals • Scalable in both quantity and rate • Metadata (# of files and transactions/sec) • Data throughput(amount of data and throughput) • Reliable • Secure • Accessible

3. What makes L-Store different?Based on a highly generic abstract block for storage(IBP)

4. LoCI ToolsLogistical Computing and Internetworking Lab • IBP Internet Backplane Protocol • Middleware for managing and using remote storage • Allows advanced space and TIME reservation • Supports multiple threads/depot • User configurable block size • Designed to support large scale, distributed systems • Provides global “malloc()” and “free()” • End-to-end guarantees • AES encryption with each allocation having a separate key • Capabilities • Each allocation has separate Read/Write/Manage keys IBP is at the “waist of the hourglass for storage” http://loci.cs.utk.edu

5. What makes L-Store different?Based on a highly generic abstract block for storage(IBP)

6. exNodeXML file containing metadata • Analogous to a disk I-node and contains • Allocations • How to assemble file • Fault tolerance encoding scheme • Encryption keys IBP Depots Network 0 100 200 300 A B C Replicated at different sites Replicated and striped Normal file

7. 3 GB/s 30 Mins L-Store Performance • Multiple simultaneous writes to 24 depots. • Each depot is a 3 TB disk server in a 1U case. • 30 clients on separate systems uploading files. • Rate has scaled linearly as depots added. • Planned REDDnet deployment of 167 Depots will be able to sustain 25 GBytes/s

8. Data Sharing 1. Sample created in lab Research Laboratory Network Analysis Laboratory Building Computing Center

9. Data Sharing 2. Taken for analysis Research Laboratory 4. Update DB 3. Store data in L-Store 1 1 Network Analysis Laboratory Building 3. Update metadata Computing Center FileA -> 1

10. Data Sharing Research Laboratory 1 1 Network Analysis Laboratory Building 5. Make copy in data center with full fault tolerance Computing Center 2 FileA -> 1 FileA -> 1,2 2 6. Which is used on the cluster 2 2

11. Data Sharing Research Laboratory 5. Researcher Analyzes data in lab 1 1 Network Analysis Laboratory 3 Building 6. Triggers local cached copy Computing Center 2 FileA -> 1,2 FileA -> 1,2,3 2 2 2

12. What is L-Store? • Provides a file system interface to (globally) distributed storage devices (“depots”) • Parallelism for high performance and reliability • Data and Metadata scale independently • Infrastructure added as needed • Uses IBP (from UTK) for data transfer & storage service. • Write: break file into blocks, upload blocks simultaneously to multiple depots (reverse for reads) • Generic, high performance, wide area capable, storage virtualization service • L-Store utilizes a DHT implementation to provide metadata scalability and reliability • Multiple metadata servers increase performance and fault tolerance • Real time addition/deletion of metadata server nodes allowed • Nevoa Networks for user interface and “LUNs” • Nevoa Explorer - WebDAV, CIFS • StorCore - Resource Management (LUNS) • L-Store supports Weaver Erasure Encoding of stored files (similar to RAID) for reliability and fault tolerance (support for up to 10 depot failures). • Can recover files even if multiple depots fail. • Computation on storage element • Support for 3rd party pluggable modules or services • File system interface • Auth/AuthZ • Flexible role based AuthZ