High Performance Data Mining Chapter 3: Clustering

1. High Performance Data MiningChapter 3: Clustering Vipin Kumar Army High Performance Computing Research Center Department of Computer Science University of Minnesota http://www.cs.umn.edu/~kumar

2. Chapter 3: Clustering Algorithms Outline • K-means Clustering • Hierarchical Clustering • Single Link, Group Average, CURE, Chameleon • Graph Based Clustering • Miscellaneous Topics • Scalability • Other Clustering Techniques

3. Clustering Definition • Given a set of data points, each having a set of attributes, and a similarity measure among them, find clusters such that • Data points in one cluster are more similar to one another. • Data points in separate clusters are less similar to one another. • Similarity Measures: • Euclidean Distance if attributes are continuous. • Other Problem-specific Measures.

4. Input Data for Clustering • A set of N points in an M dimensional spaceOR • A proximity matrix that gives the pairwise distance or similarity between points. • Can be viewed as a weighted graph.

5. K-means Clustering • Find a single partition of the data into K clusters such that the within cluster error, e.g., , is minimized. • Basic K-means Algorithm: 1. Select K points as the initial centroids. 2. Assign all points to the closest centroid. 3. Recompute the centroids. 4. Repeat steps 2 and 3 until the centroids don’t change. • K-means is a gradient-descent algorithm that always converges - perhaps to a local minimum. (Clustering for Applications, Anderberg)

6. Example: Kmeans Initial Data and Seeds Final Clustering

7. Example: K-means Initial Data and Seeds Final Clustering

8. K-means: Initial Point Selection • Bad set of initial points gives a poor solution. • Random selection • Simple and efficient. • Initial points don’t cover clusters with high probability. • Many runs may be needed for optimal solution. • Choose initial points from • Dense regions so that the points are “well-separated.” • Many more variations on initial point selection.

9. K-means: How to Update Centroids • Depends on the exact error criterion used. • If trying to minimize the squared error, , then the new centroid is the mean of the points in a cluster. • If trying to minimize the sum of distances,, then the new centroid is the median of the points in a cluster.

10. K-means: When to Update Centroids • Update the centroids only after all points are assigned to centers. • Update the centroids after each point assignment. • May adjust the relative weight of the point being added and the current center to speed convergence. • Possibility of better accuracy and faster convergence at the cost of more work. • Update issues are similar to those of updating weights for neural-nets using back-propagation. (Artificial Intelligence, Winston)

11. K-means: Pre and Post Processing • Outliers can dominate the clustering and, in some cases, are eliminated by preprocessing. • Post-processing attempts to “fix-up” the clustering produced by the K-means algorithm. • Merge clusters that are “close” to each other. • Split “loose” clusters that contribute most to the error. • Permanently eliminate “small” clusters since they may represent groups of outliers. • Approaches are based on heuristics and require the user to choose parameter values.

12. K-means: Time and Space requirements • O(MN) space since it uses just the vectors, not the proximity matrix. • M is the number of attributes. • N is the number of points. • Also keep track of which cluster each point belongs to and the K cluster centers. • Time for basic K-means is O(T*K*M*N), • T is the number of iterations. (T is often small, 5-10, and can easily be bounded, as few changes occur after the first few iterations).

13. K-means: Determining the Number of Clusters • Mostly heuristic and domain dependant approaches. • Plot the error for 2, 3, … clusters and find the knee in the curve. • Use domain specific knowledge and inspect the clusters for desired characteristics.

14. K-means: Problems and Limitations • Based on minimizing within cluster error - a criterion that is not appropriate for many situations. • Unsuitable when clusters have widely different sizes or have convex shapes. • Restricted to data in Euclidean spaces, but variants of K-means can be used for other types of data.

15. K-medoid Clustering • Find a single partition of the data into K clusters such that each cluster has a most representative point, i.e., a point that is the most “centrally” located point in the cluster with respect to some measure, e.g., distance. • These representative points are called medoids. • Basic K-medoid Algorithm: 1. Select K points as the initial medoids. 2. Assign all points to the closest medoid. 3. See if any other point is a “better” medoid. 4. Repeat steps 2 and 3 until the medoids don’t change.

16. K-medoid Clustering • Can be used with similarities, as well as distances and there is no Euclidean space restriction. • Finding a better medoid involves comparing all pairs of medoid and non-medoid points and is relatively inefficient. • Sampling may be used.(Efficient and effective clustering method for spatial data mining. Ng and Han, ‘94) • Better resistance to outliers.(Finding Groups in Data, Kaufman and Rousseeuw)

17. Types of Clustering: Partitional and Hierarchical • Partitional Clustering ( K-means and K-medoid) finds a one-level partitioning of the data into K disjoint groups. • Hierarchical Clustering finds a hierarchy of nested clusters (dendogram). • May proceed either bottom-up (agglomerative) or top-down (divisive). • Uses a proximity matrix. • Can be viewed as operating on a proximity graph.

18. Hierarchical Clustering Algorithms • Hierarchical Agglomerative Clustering1. Initially each item belongs to a single cluster.2. Combine the two most similarclusters.3. Repeat step 2 until there is only a single cluster. • Most popular approach. • Hierarchical Divisive Clustering • Starting with a single cluster, divide clusters until only single item clusters remain. • Less popular, but equivalent in functionality.

19. 1 2 3 4 5 Cluster Similarity: MIN or Single Link • Similarity of two clusters is based on the two most similar (closest) points in the different clusters. • Determined by one pair of points, i.e., by one link in the proximity graph. • Can handle non-elliptical shapes. • Sensitive to noise and outliers.

20. 1 2 3 4 5 Cluster Similarity: MAX or Complete Linkage • Similarity of two clusters is based on the two least similar (most distant) points in the different clusters. • Determined by all pairs of points in the two clusters. • Tends to break large clusters. • Less susceptible to noise and outliers.

21. 1 2 3 4 5 Cluster Similarity: Group Average • Similarity of two clusters is the average of pairwise similarities between points in the two clusters. • Compromise between Single and Complete Link. • Need to use average connectivity for scalability since total connectivity favors large clusters.

22. Cluster Similarity: Centroid Methods • Similarity of two clusters is based on the distance of the centroids of the two clusters. • Similar to K-means • Euclidean distance requirement • Problems with different sized clusters and convex shapes. • Variations include “median” based methods.

23. Hierarchical Clustering: Time and Space requirements • O(N2) space since it uses the proximity matrix. • N is the number of points. • O(N3) time in many cases. • There are N steps and at each step the size, N2,proximity matrix must be updated and searched. • By being careful, the complexity can be reduced to O(N2 log(N) ) time for some approaches.

24. Hierarchical Clustering: Problems and Limitations • Once a decision is made to combine two clusters, it cannot be undone. • No objective function is directly minimized. • Different schemes have problems with one or more of the following: • Sensitivity to noise and outliers. • Difficulty handling different sized clusters and convex shapes. • Breaking large clusters.

25. Recent Approaches: CURE • Uses a number of points to represent a cluster. • Representative points are found by selecting a constant number of points from a cluster and then “shrinking” them toward the center of the cluster. • Cluster similarity is the similarity of the closest pair of representative points from different clusters. • Shrinking representative points toward the center helps avoid problems with noise and outliers. • CURE is better able to handle clusters of arbitrary shapes and sizes.(CURE, Guha, Rastogi, Shim)

26. Experimental ResultsCURE (centroid) (single link) Picture from CURE, Guha, Rastogi, Shim.

27. Experimental ResultsCURE (centroid) (single link) Picture from CURE, Guha, Rastogi, Shim.

28. Limitations of Current Merging Schemes • Existing merging schemes are static in nature.

29. Chameleon: Clustering Using Dynamic Modeling • Adapt to the characteristics of the data set to find the natural clusters. • Use a dynamic model to measure the similarity between clusters. • Main property is the relative closeness and relative inter-connectivity of the cluster. • Two clusters are combined if the resulting cluster shares certain properties with the constituent clusters. • The merging scheme preserves self-similarity. • One of the areas of application is spatial data.

30. Characteristics of Spatial Data Sets • Clusters are defined as densely populated regions of the space. • Clusters have arbitrary shapes, orientation, and non-uniform sizes. • Difference in densities across clusters and variation in density within clusters. • Existence of special artifacts (streaks) and noise. The clustering algorithm must address the above characteristics and also require minimal supervision.

31. Chameleon: Steps • Preprocessing Step: Represent the Data by a Graph • Given a set of points, we construct the k-nearest-neighbor (k-NN)graph to capture the relationship between a point and its k nearest neighbors. • Phase 1: Use a multilevel graph partitioning algorithm on the graph to find a large number of clusters of well-connected vertices. • Each cluster should contain mostly points from one “true” cluster, i.e., is a sub-cluster of a “real” cluster. • Graph algorithms take into account global structure.

32. Chameleon: Steps … • Phase 2: Use Hierarchical Agglomerative Clustering to merge sub-clusters. • Two clusters are combined if the resulting cluster shares certain properties with the constituent clusters. • Two key properties are used to model cluster similarity: • Relative Interconnectivity: Absolute interconnectivity of two clusters normalized by the internal connectivity of the clusters. • Relative Closeness: Absolute closeness of two clusters normalized by the internal closeness of the clusters.

33. Experimental ResultsCHAMELEON

34. Experimental ResultsCHAMELEON

35. Experimental ResultsCURE (10 clusters)

36. Experimental ResultsCURE (15 clusters)

37. Experimental ResultsCHAMELEON

38. Experimental ResultsCURE (9 clusters)

39. Experimental ResultsCURE (15 clusters)

40. Hierarchical Divisive Clustering • Starting with a single cluster, divide clusters until only single item clusters remain. • MST (Minimum Spanning Tree). • Same as Single Link. • Susceptible to noise and outliers. • Graph-based clustering. • Global view. • Less susceptible to noise and outliers. • Example: Graph-base clusteringis not fooled by “bridges”.

41. Hypergraph-Based Clustering Construct a hypergraph in which related data are connected via hyperedges. Partition this hypergraph in a way such that each partition contains highly connected data. How do we find related sets of data items? Use Association Rules!

42. S&P 500 Stock Data • S&P 500 stock price movement from Jan. 1994 to Oct. 1996. • Frequent item sets from the stock data. Day 1: Intel-UP Microsoft-UP Morgan-Stanley-DOWN … Day 2: Intel-DOWN Microsoft-DOWN Morgan-Stanley-UP … Day 3: Intel-UP Microsoft-DOWN Morgan-Stanley-DOWN …  {Intel-up, Microsoft-UP} {Intel-down, Microsoft-DOWN, Morgan-Stanley-UP} {Morgan-Stanley-UP, MBNA-Corp-UP, Fed-Home-Loan-UP} 

43. Clustering of S&P 500 Stock Data Other clusters found: Bank, Paper/Lumber, Motor/Machinery, Retail, Telecommunication, Tech/Electronics

44. 1984 Congressional Voting records • Voting records on 16 key votes of 435 congressmen • Frequent item sets from the voting data. Congressman 1: crime-YES, education-spending-YES, mx-missile-NO, … Congressman 2: crime-YES,education-spending-NO,mx-missile-YES, ... Congressman 3: crime-NO,education-spending-NO,mx-missile-YES, …  {crime-YES, education-spending-YES} {education-spending-NO, mx-missile-YES} {crime-NO, education-spending-NO, physician-fee-freeze-NO} 

45. Clustering of Congressional Voting Data

46. Clustering of ESTs in Protein Coding Database Laboratory Experiments New Protein Functionality of the protein Similarity Match Researchers John Carlis John Riedl Ernest Retzel Elizabeth Shoop Clusters of Short Segments of Protein-Coding Sequences (EST) Known Proteins

47. Expressed Sequence Tags (EST) • Generate short segments of protein-coding sequences (EST). • Match ESTs against known proteins using similarity matching algorithms. • Find Clusters of ESTs that have same functionality. • Match new protein against the EST clusters. • Experimentally verify only the functionality of the proteins represented by the matching EST clusters

48. EST Clusters by Hypergraph-Based Scheme • 662 different items corresponding to ESTs. • 11,986 variables corresponding to known proteins • Found 39 clusters • 12 clean clusters each corresponds to single protein family (113 ESTs) • 6 clusters with two protein families • 7 clusters with three protein families • 3 clusters with four protein families • 6 clusters with five protein families • Runtime was less than 5 minutes.

49. EST clusters by LSI/K-means • Dimension has been reduced to 50 using LSI • Found 38 clusters • 17 clean clusters (69 ESTs) • 8 clusters with several protein families • 1 cluster with 508 ESTs • 22 clusters with one or two ESTs

50. Web Document Data • Explosive growth of documents on the World Wide Web • Searching relevant documents becomes a challenge • Cluster of related words from documents can serve as keywords in searching • Cluster of documents can aid filtering and categorization of retrieved web documents