1 / 24

Machine Learning in DryadLINQ

This document explores the integration of DryadLINQ with large vector libraries for efficient data analysis and machine learning on distributed systems. We delve into how the software stack simplifies programming for large clusters using declarative programming, allows various operations on large vectors, and supports complex algorithms like linear regression and expectation-maximization. With practical examples and code, we demonstrate the effectiveness of these tools for analyzing large-scale datasets, including insights into botnet traffic.

quinlan-rutledge
Télécharger la présentation

Machine Learning in DryadLINQ

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Machine Learning in DryadLINQ Kannan Achan Mihai Budiu MSR-SVC, 1/30/2008

  2. Goal

  3. The Software Stack Data analysis Machine learning Large Vector DryadLINQ Dryad Distributed Filesystem: Cosmos Cluster Services Windows Server Windows Server Windows Server

  4. Dryad

  5. Dryad Jobs Input files R R R R Stage X X X X X X M M M M Vertices (processes) Channels M M Output files

  6. LINQ and C#

  7. LINQ Collection<T> collection; boolIsLegal(Key); string Hash(Key); var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};

  8. DryadLINQ = LINQ + Dryad Collection<T> collection; boolIsLegal(Key k); string Hash(Key); var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value}; Vertexcode Queryplan (Dryad job) Data collection C# C# C# C# results

  9. Recall: The Software Stack Data analysis Machine learning Large Vector DryadLINQ Dryad Distributed Filesystem: Cosmos Cluster Services Windows Server Windows Server Windows Server

  10. Very Large Vector Library PartitionedVector<T> T T T Scalar<T> T

  11. Operations on Large Vectors: Map 1 T f U f preserves partitioning T f U

  12. Map 2 (Pairwise) T f U V T U f V

  13. Map 3 (Vector-Scalar) T f U V T U f V 13

  14. Reduce (Fold) f U U U U f f f U U U f U

  15. Linear Algebra T T V = U , ,

  16. Linear Regression • Data • Find • S.t.

  17. Analytic Solution X[0] X[1] X[2] Y[0] Y[1] Y[2] Map X×XT X×XT X×XT Y×XT Y×XT Y×XT Reduce Σ Σ [ ]-1 * A

  18. Linear Regression Code Matrices xx = x.PairwiseOuterProduct(x); OneMatrixxxs= xx.Sum(); Matrices yx = y.PairwiseOuterProduct(x); OneMatrixyxs= yx.Sum(); OneMatrixxxinv = xxs.Map(a => a.Inverse()); OneMatrix A = yxs.Map(xxinv, (a, b) => a.Multiply(b));

  19. Expectation Maximization • 160 lines • 3 iterations shown

  20. Understanding Botnet Traffic using EM • 3 GB data • 15 clusters • 60 computers • 50 iterations • 9000 processes • 50 minutes

  21. Conclusions • Dryad simplifies programming large clusters • DryadLINQ = declarative programming for Dryad jobs • The Large Vector library provides simple mathematical primitiveson top of DryadLINQ • Matlab-style coding for writing distributed numeric computations Data analysis ML Large Vector DryadLINQ Dryad Distributed Filesystem Cluster Services Win Win Win

  22. Backup Slides

  23. Chaining X[0] X[1] X[2] Y[0] Y[1] Y[2] X×XT X×XT X×XT Y×XT Y×XT Y×XT Σ Σ Σ Σ Σ Σ Σ Σ [ ]-1 * A

  24. EM Structure E stage π μ σ Input size All parameters

More Related