1 / 14

The Storage

The Storage. B. Ramamurthy. Topics for discussion. On chip memory On board memory System memory Off system/online storage/ secondary memory File system abstraction Offline/ tertiary memory RAID: Redundant Array of Inexpensive Disks NAS: Network Accessible Storage

kaitlin-hays
Télécharger la présentation

The Storage

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Storage B. Ramamurthy C B. Ramamurthy

  2. Topics for discussion • On chip memory • On board memory • System memory • Off system/online storage/ secondary memory • File system abstraction • Offline/ tertiary memory • RAID: Redundant Array of Inexpensive Disks • NAS: Network Accessible Storage • SAN: Storage area networks • DB and DBMS: Data base and DB management systems • Distributed file system • Google file system • Hadoop file system C B. Ramamurthy

  3. Data and Computation Continuum Compute intensive Ex: computation of digits of PI Data intensive Ex: analyzing web logs C B. Ramamurthy

  4. More dimensions Other variables: Communication Bandwidth, ? PFLOPS Massively Multiplayer Online game (MMOG) Realtime Systems TFLOPS Digital Signal Processing Compute scale Business Analytics GFLOPS Weblog Mining MFLOPS Payroll K M G T P Data scale C B. Ramamurthy

  5. Solution Processing Granularity Data size: small Pipelined Instruction level Concurrent Thread level Service Object level Indexed File level Mega Block level Virtual System Level Data size: large

  6. On chip memory • Registers • Cache • Buffers (instruction pipeline) • Characteristics: volatile C B. Ramamurthy

  7. On board memory • Cache • Instructions cache • Data cache • Translation look aside buffers (TLB) • Characteristics: content addressable, set-associative organization C B. Ramamurthy

  8. System memory C B. Ramamurthy

  9. Off-system storage (Earlier Lectures covered these) C B. Ramamurthy

  10. Database and Database Management System • Data source • Transactional • Data base server • Relational db or similar foundation • Tables, rows, result set, SQL • ODBC: open data base connectivity • Very successful business model: Oracle, DB2, MySQL, and others • Persistence models: EJB, DAO, ADO (I am not going to expand the abbreviation.. ) C B. Ramamurthy

  11. Distributed file system(DFS) • A dedicated server manages the files for an compute environment • For example, nickelback,cse.buffalo.edu is your file server and that is why we did not want you to run your user applications on this machine. • DFS addresses various transparencies: location transparency, sharing, performance etc. • Examples: NFS, NFS+, AFS (Andrew FS)… (you will study these in Distributed Systems course) C B. Ramamurthy

  12. On to Google File • Internet introduced a new challenge in the form web logs, web crawler’s data: large scale “peta scale” • But observe that this type of data has an uniquely different characteristic than your transactional or the “order” data on amazon.com: “write once” ; so is HIPPA protected healthcare and patient information; • Google exploited this characteristics in its Google file system: S. Ghemavat C B. Ramamurthy

  13. Hadoop File System (HFS) • Hadoop file system is a reverse engineered version of the GFS : this is my first opinion on HFS • HFS is a distributed file system for large scale data • Data throughput is more important than latency • Batch computing than interactive time shared computing C B. Ramamurthy

  14. MapReduce Cat Bat Dog Other Words (size: TByte) reduce combine map part0 split reduce combine map part1 split reduce combine map split part2 map split

More Related