1 / 27

Resource Allocation for Distributed Streaming Applications

Resource Allocation for Distributed Streaming Applications. Qian Zhu and Gagan Agrawal Department of Computer Science and Engineering The Ohio State University. ICPP 2008 Conference. Sept. 10 th , 2008 Portland, Oregon. ICPP 2008. Data Streaming Applications. Computational Steering

pink
Télécharger la présentation

Resource Allocation for Distributed Streaming Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Resource Allocation for Distributed Streaming Applications Qian Zhu and Gagan Agrawal Department of Computer Science and Engineering The Ohio State University ICPP 2008 Conference Sept. 10th, 2008 Portland, Oregon ICPP 2008

  2. Data Streaming Applications • Computational Steering • Interactivelycontrol scientific simulations • Computer Vision Based Surveillance • Track people and monitor critical infrastructure • Images captured by multiple cameras • Online Network Intrusion Detection • Analyze connection request logs • Identify unusual patterns 2 ICPP 2008

  3. Streaming Applications in Wide Area Environments • Distributed high-volume data sources • Increasing WAN bandwidths • Better than secondary storage bandwidths • Geographically distributed users / consumers of data • Exploit flexibility in resource usage in Grid Environments

  4. Our Previous Work • A middleware system GATES • Grid-based AdapTive Executions on Streams • Integration with Grid Standards • Support for self-adaptation • Dynamic allocation and fault-tolerance

  5. Resource Allocation in Streaming Grid Applications • Challenges • Pipeline of processing stages • Computation and communication requirements • Long running nature • Dynamic grid resources • Current Approach • Ad Hoc and Heuristics-based • Not considering both bandwidth and computing power ICPP 2008

  6. Overview of Our Research • Static Resource Allocation • Subgraph isomorphism based • Handle Network bandwidth and Computing power • Effectiveness value • Goal • To minimize the execution time of the data streaming applications ICPP 2008

  7. Outline Motivation and Introduction Resource Allocation in Data Stream Processing Resource Allocation Algorithm Experimental Evaluation Related Work Conclusion 7 ICPP 2008

  8. Data Stream Processing Model • Directed Acyclic Graph (DAG) – Gp(Vp, Ep) source Computing Power Requirement Bandwidth Requirement Processing nodes sink ICPP 2008

  9. Resource Model • Directed Acyclic Graph (DAG) – GR(VR, ER) Computing power Bandwidth ICPP 2008

  10. Problem Description • To Allocate Resources to the Data Stream Application • A mapping from Gp(Vp, Ep) to GR(VR, ER) • Modified Subgraph Isomorphism Based • To choose an isomorphic subgraph of GR • Transporters • Optimal Mapping • Effectiveness value • To minimize the execution time ICPP 2008

  11. Example A 1000 B 400 C 200 E 2000 D 100 transporter ICPP 2008

  12. Effectiveness Value • Bandwidth only • Including Computing Power Number of transporters A sigmoid function Overhead of adding transporters Computing power match ICPP 2008

  13. Outline Motivation and Introduction Resource Allocation in Data Stream Processing Resource Allocation Algorithm Experimental Evaluation Related Work Conclusion 13 ICPP 2008

  14. Proposed Algorithm • Background: VF algorithm (L.P.Cordella et al.) • State Space Representation (SSR) • Feasibility rules • Depth-First Search • Pros and Cons • Efficient with small graphs (<200 nodes) • A large number of candidate partial mappings ICPP 2008

  15. 1 2 3 4 Proposed Algorithm – Step 1 • Prune Candidate Partial Mappings • Candidate node list • Reduce potential matches • Multiple Partial Mapping set A B C D E F G C D G E F A 1000 3 200 Cand(3)={C,D,G} B 400 C 200 E 2000 D 100 ICPP 2008

  16. 1 2 3 4 Proposed Algorithm – Step 2 • Modified Subgraph Isomorphism Mapping • Transporters A B C D E F G C D G E F A 1000 3 200 B 400 C 200 E 2000 D 100 Candidate pair: (3,C) transporter Candidate pair: (3,D) ICPP 2008

  17. Handle Computing Power • Computing Node  Network Link • Computing power  Network bandwidth • Effectiveness Value Calculation • Possible Issues: high bandwidth and low computing power • Map one node onto a cluster of network nodes ICPP 2008

  18. Outline Motivation and Introduction Resource Allocation in Data Stream Processing Resource Allocation Algorithm Experimental Evaluation Related Work Conclusion 18 ICPP 2008

  19. Goals for the Experiments • Demonstrate the Scalability of Our Resource Allocation Algorithm • Demonstrate the High Performance of the Applications ICPP 2008

  20. Experiment Setup • Algorithms Compared • Optimal • Streamline • Streaming Applications • Volume Rendering Application • A Synthetic Application ICPP 2008

  21. Scalability of the Resource Allocation Algorithm ICPP 2008

  22. Application Performance • Volume Rendering 33% 29% 27% Within 4% ICPP 2008

  23. Application Performance • A Synthetic Application 40% 36% Within 3% 34% ICPP 2008

  24. Outline Motivation and Introduction Resource Allocation in Data Stream Processing Resource Allocation Algorithm Experimental Evaluation Related Work Conclusion 24 ICPP 2008

  25. Related Work • Resource Allocation for Stream Processing • Tang et al. (HPCC 06), Ali et al. (PDPTA 02) • Resource Allocation for Grid Computing • Abdu et al. (IPDPS 01), Bhat et al. (Grid 07), Hong et al. (ICPP 03) • Subgraph Isomorphism Algorithms and Applications • Bioinformatics (Online Information 90), VLSI design (ISCAS 95), Mobile robot design (JPR 95) ICPP 2008

  26. Conclusion • Modified Subgraph Isomorphism Algorithm for Resource Allocation in Grid Streaming Applications • Handling Network Bandwidth and Computing Power • Comparable Overhead with Streamline • Improved Application Performance ICPP 2008

  27. Thank you! ICPP 2008

More Related