1 / 39

Efficient and Flexible Parallel Retrieval using Priority Encoded Transmission(2004)

Efficient and Flexible Parallel Retrieval using Priority Encoded Transmission(2004). CMPT 886 Represented By: Lilong Shi. Introduction. Motivations Multiple servers Increase of scalability Load-balancing by multiple servers Efficient multicast-based data distribution Peer-to-peer systems

kylee-vang
Télécharger la présentation

Efficient and Flexible Parallel Retrieval using Priority Encoded Transmission(2004)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Efficient and Flexible Parallel Retrieval using Priority Encoded Transmission(2004) CMPT 886 Represented By: Lilong Shi

  2. Introduction • Motivations • Multiple servers • Increase of scalability • Load-balancing by multiple servers • Efficient multicast-based data distribution • Peer-to-peer systems • Parallel downloading from multiple peers

  3. Problem and Existing Solutions • Concern • Server: scalability and fault-tolerance • Client: download rate and avoid duplicates from multiple servers • Current Solutions • Client requests disjoint portions • High cost • In P2P, no centralized control

  4. Problem and Existing Solutions • Current Solutions • Encoded transmission • Server encodes the whole document • Client downloads enough encoded distinct data items that enables it to recover the original content • Disadvantage: • high cost for encoding and decoding • No flexible access to structured document

  5. Proposed Solution • Goal • To achieve efficient downloads by coordinating the download process with each server in a manner that optimizes the servers’ capabilities with the clients’ requirement • Solution • Parallel retrieval using priority encoded transmission • Flexibility • Efficiency • Loss-Resilience

  6. Sections • Parallel Retrieval • Retrieving from multiple resources • Rate Allocation • Optimal bandwidth to minimize the downloading time • Additive Transmission • PET approach – lossy transmission • Applications • Bulk distribution and streaming distribution

  7. Sections • Parallel Retrieval • Retrieving from multiple resources • Rate Allocation • Optimal bandwidth to minimize the downloading time • Additive Transmission • PET approach – lossy transmission • Applications • Bulk distribution and streaming distribution

  8. Parallel Retrieval - The Abstract Model • Notations: • Symbols are data units of fixed size • A channel C has a rate |C| symbols per unit time • A page P of size |P| transmitted with rate r <= |C| • A client is described as <D, C, p> • D is a document to be downloaded • C is a channel available to the client • p is a schedule of pages in D, the download rate of each page • A server is described as <X, C> • X is a set of pages stored in the server • C is the channel available to the server

  9. Parallel Retrieval - The Abstract Model • Download process of a client <D, C, p> • 3 phases: location, initiation, and transmission • Step1 - Location • Not discussed in this paper • Assume client locates a set of servers for D S = {<Xi, Ci> : i = 1 to |S|} • Each server Si can transmit at rate |Ci|

  10. Parallel Retrieval - The Abstract Model • Step 2 – Initiation • Each server Si sends a description <Li, |Ci|> to client - Li contains a list of indices of every page Pj Xi • Based on the description <Li, |Ci|> and its schedule p, the client computes a rate vector R = {rij} for each page at each server • i is server ID, and j is page ID • Client sends a req Ri = {j, rij} to server Si

  11. Parallel Retrieval - The Abstract Model • Step3 – Transmission • Each server Si, upon receiving a request Ri, transmits a message Mi on Ci to the client. • Mi is encoded using PET, which will be discussed in the next section

  12. Parallel Retrieval - The Abstract Model • An example • D is an 6 MB file with 3 pages P1, P2 and P3 each of size 2MB. The bandwidth of servers S1 and S2 are |C1| = 200 kbps, and |C2| = 300 kbps. S1 contains P1 and P2, S2 contains P2 and P3. • A client <D, C> with |C| = 500 kbps attempts to download D from both S1 and S2 with schedule p = <500/3, 500/3, 500/3>

  13. Parallel Retrieval - The Abstract Model • Naïve solution • S1 and S2 both split their bandwidths equally between their two pages, the aggregate download rates are 100kpbs for P1, 250 kbps for P2 and 150 kbps for P3. That is, Client sends request R1 = {(1, 100), (2, 100)} to S1, and R2 = {(2, 150),(3, 150)} to S2. The entire download time = 16000/100 = 160 seconds. • Ideal solution • Client sends request R1 = {(1, 500/3), (2, 100/3)} to S1, and R2 = {(2, 400/3),(3, 500/3)} to S2, so that S1 allocates 5/6 of its bandwidth for P1, 1/6 for P2; S2 allocates 4/9 of its bandwidth for P2, 5/9 for P3. Then all pages can be downloaded in optimal 96 secs

  14. Parallel Retrieval - Solve the Model • Problems to be solved • Rate Allocation: • During initiation, given descriptions <Li, |Ci|> from each server, and its schedule p, the client should compute a rate vector R that will enable it to meet its download goals. • Information-Additive Transmission: - During transmission, each server Si, by sending an appropriately encoded message Mi of its content Xi, should enable the client to gather messages from all serves and fulfill its download schedule p for the entire document D

  15. Sections • Parallel Retrieval • Retrieving from multiple resources • Rate Allocation • Optimal bandwidth to minimize the downloading time • Additive Transmission • PET approach – lossy transmission • Applications • Bulk distribution and streaming distribution

  16. Rate Allocation – Optimal solution • Problem definition • A client <D, C, p> allocating rates for each page Pi D to each server Si  S to meet its download schedule p. • The problem consists of determining a set of flows rij, such that the inflow at any sink Pi (the aggregate download rate of page Pi from all servers) meets the demand p(Pi) (the download rate required for Pi by the schedule) without the outflow at any sources Si (the rate of transmission at server Si) exceeding its supply |Ci| (the channel bandwidth of Si). • A linear programming problem

  17. Rate Allocation – Feasible solution • Observations and Facts • Demand at each sink Pj cannot always be met through its inflow pj while simultaneously restricting the outflow i at each source Si to be no greater than its supply |Ci| • Simply speaking, it’s not always possible that client meets the schedule with the restriction of servers’ bandwidth

  18. Rate Allocation – Feasible solution • Feasible solution • This optimization problem is to find a schedule that minimizes the maximum cost of a client Minimize max {CPj(pj), j = 1, 2, ….,|D|} Subject to i <= |Ci|, i=1,2,….,|S| This problem can be solved as a linearly constrained mini-max optimization .

  19. Sections • Parallel Retrieval • Retrieving from multiple resources • Rate Allocation • Optimal bandwidth to minimize the downloading time • Additive Transmission • PET approach – lossy transmission • Applications • Bulk distribution and streaming distribution

  20. Additive Transmission • Problems during transmission • Lossy network, some symbols may be dropped • May transmit duplicate symbols from multiple servers • Solution • Priority Encoded Transmission (PET)

  21. Additive Transmission - PET • PET Review • Given a message of m symbols, a PET system produces n encoded symbols, where m out of the n encoded symbols can be used to reconstruct the message.

  22. Additive Transmission - PET • PET as a solution • A server S encodes each page Pj into Nj symbols • Server S then constructs n packets of length L by choosing kj out of nj(≤Nj) encoded symbols per page Pj for each packet

  23. Additive Transmission - PET • PET as a solution • Client should not merely be able to reconstruct page Pj from |Pj| distinct encoded symbols, but also from |Pj| distinct encoded symbols received from multiple servers. • This can be achieved by the client computing Nj and partitioning it into appropriately sized regions for each server.

  24. Sections • Parallel Retrieval • Retrieving from multiple resources • Rate Allocation • Optimal bandwidth to minimize the downloading time • Additive Transmission • PET approach – lossy transmission • Applications • Bulk distribution and streaming distribution

  25. Applications – Bulk Distribution • Bulk distribution • Ideally, for client • This can be achieved by choosing a schedule in which all pages finish downloading at time tmin • Thus, for bulk data distribution the client picks a schedule p

  26. Applications – Bulk Distribution • For infeasible solution • The total downloading time is determined by the slowest or bottleneck page to download • If the inflow (downloading rate) of page Pj is xj, then Pj can be downloaded in time |Pj|/xj => The cost function for each page • So we want to solve for every xj

  27. Application – Bulk Distribution • Review • This optimization problem is to find a schedule that minimizes the maximum cost of a client Minimize max {CPj(pj), j = 1, 2, ….,|D|} Subject to i <= |Ci|, i=1,2,….,|S| This problem can be solved as a linearly constrained mini-max optimization .

  28. constant Applications – Bulk Distribution • Consider the example before • And • The MinMax problem can be formed as Minimize max {2MB/r11, 2MB/(r12+r22), 2MB/r23} Subject to constraints: r11 + r12 <= 200, r22 + r23 <= 300, r11+r12+r22+r23 <= 500, r11, r12, r22, r23 >=0 Solution is r11 = 500/3, r12 = 100/3, r22 = 400/3, r23 = 500/3 or, x1 = x2 = x3 = 500/3

  29. Applications – Streaming Distribution • Streaming distribution • Assume asynchronous multicast is used • Assume after beginning of download process, a client has to wait for time w before playback • Assume the playback rate is 1 symbol/unit time • Thus P1 is recoverable in time w, P2 in time w+|P1|,…., and in general Pj in w + |Pj-1|

  30. Applications – Streaming Distribution • Infeasible solution • When server rates are constrained, some pages may take longer than p(Pj) to download. • This causes a longer delay before playback. ie. w. Therefore, the client has to wait for time w’ > w such that the download time for Pj becomes w’ + |Pj-1| • The cost function is just w’-w

  31. constant Applications – Streaming Distribution • Consider the example before • And • The MinMax problem can be formed as Minimize max {2MB/r11, 2MB/(r12+r22)-2MB, 2MB/r23-4MB} Subject to constraints: r11 + r12 <= 200, r22 + r23 <= 300, r11+r12+r22+r23 <= 500, r11, r12, r22, r23 >=0 Solution is r11 = 200, r12 = 0, r22 = 136, r23 = 68 or, x1=200, x2=136, x3=68 => w=t0=80, t1=118, t2=235, t38

  32. Performance • Assuming file-sharing information is available • i.e. Location step is not necessary • Setups • 1 GB file divided into 100 pages • For bulk download, the goal is to download in the shortest time • For streaming case, assume a 8000 seconds of 1 MB video. • Randomly select servers from the entire list • Randomly distribute documents to servers

  33. Performance • Parameters • Server slack (s) is the ratio of randomly selected servers to the number of actual downloads initiated. • i.e. to select N servers, we randomly choose sxN servers and selected the N ones with most bandwidth • Page slack (s) is the ratio of pages of D (duplicates) stored in these servers to the size of the document D • The plot is the on average over 100 runs

  34. Performance - Bulk

  35. Performance - Streaming

  36. Performance - Analysis • Downloading time shrunk by a factor 3-7 on an average as the number of servers goes from 1 – 16 • For a given number of servers, the performance increases with increasing server slack s and page slack p

  37. Conclusions • Proposed a way to determine the optimal priorities to enable a client to meet its schedule • MinMax the cost function • Efficient download from multiple sources while optimizing the bandwidth usage of servers • Increased scalability and fault-tolerance • Used PET in a lossy network environment and avoid duplicated transmission

  38. Future Work - Adaptive protocol • Adaptive protocol • Static schedule vs. dynamic schedule • Servers react to traffic/congestions • In peer-to-peer system, peers arrive and leave • Adaptive protocol • Clients re-compute priorities (schedule) periodically based on measurements and send update messages back to the servers. • One of the server periodically initiating re-scheduling requests to all clients • Efficiency vs. overhead

  39. Reference • Ramaprabhu Janakiraman and Lihao Xu: Efficient and Flexible Parallel Retrieval using Priority Encoded Transmission, (to appear in) Proceedings of the 14th ACM International Workshop on Network and Operating Systems Support for Digital Audio and Video (ACM NOSSDAV 2004), Kinsale, Ireland, June 2004

More Related