1.05k likes | 1.07k Vues
Density-Based Data Clustering Algorithms: K-Means & Others. Jianping Fan CS Department UNC-Charlotte. http://webpages.uncc.edu/jfan/. What is Data Clustering?.
E N D
Density-Based Data Clustering Algorithms: K-Means & Others Jianping Fan CS Department UNC-Charlotte http://webpages.uncc.edu/jfan/
What is Data Clustering? • Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups
Applications of Data Clustering • Understanding • Group related documents for browsing, group genes and proteins that have similar functionality, or group stocks with similar price fluctuations • Summarization • Reduce the size of large data sets Why? Clustering precipitation in Australia
Inter-cluster distances are maximized Intra-cluster distances are minimized What are key issues for data clustering? • Similarity or distance function • Inter-cluster similarity or distance • Intra-cluster similarity or distance • Number of clusters
What are key issues for data clustering? • Similarity or distance function x z Distance Similarity function
What are key issues for data clustering? • Intra-cluster similarity or distance
What are key issues for data clustering? • Inter-cluster similarity or distance
What are key issues for data clustering? • Number of clusters K
What are key issues data clustering? • Objective function for clustering Inter-Cluster Distance
What are key issues data clustering? • Objective function for clustering Intra-Cluster Distance
Inter-cluster distances are maximized Intra-cluster distances are minimized What are key issues data clustering? • Objective function for clustering Objective Function for Clustering
K-Means (K-Centers) Clustering Cluster Centers
K-Means Clustering • Similarity or distance function x z Distance Similarity function
K-Means Clustering • Similarity or distance function for K-means clustering
K-Means Clustering • Intra-cluster similarity or distance
K-Means Clustering • Inter-cluster similarity or distance
K-Means Clustering • Number of clusters K
K-Means Clustering • Objective function for clustering Inter-Cluster Distance
K-Means Clustering • Objective function for clustering Intra-Cluster Distance
Inter-cluster distances are maximized Intra-cluster distances are minimized K-Means Clustering • Objective function for clustering Objective Function for Clustering
Inter-cluster distances are maximized Intra-cluster distances are minimized K-Means Clustering • Objective Function for K-Means Clustering
K-Means Clustering: Clustering for New Sample y Test Sample Test sample is assigned into the closest cluster!
How many clusters? Six Clusters Two Clusters Four Clusters Notion of a Cluster can be Ambiguous Why it is ambiguous?
K-Means Clustering Approach • Number of K (centers) • Similarity or distance function • Assignment rules • Determining number of centers • Putting centers in dense areas • Distance as similarity • Assign into the closest one
K-Means Clustering • What are the potential problems for K-means clustering? • Number of centers? K • Where are the centers? Center selection • Inter-cluster overlapping • Data manifold
K-Means Clustering • Number of centers? K • Where are the centers? Center selection
K-Means Clustering • Number of centers? K • Where are the centers? Center selection
K-Means Clustering • Number of centers? K • Where are the centers? Center selection If K is set too big, what we can do?
K-Means Clustering • Number of centers? K • Where are the centers? Center selection If K is set too small, what we can do?
K-Means Clustering • If initial K is too big, what we can do further? Merging the overlapped clusters as a bigger one Bottom-Up Approach
K-Means Clustering • If initial K is too small, what we can do further? Top-Down Approach Separating diverse cluster into multiple homogeneous ones
K-Means Clustering • Where are the centers? Center selection How to identify dense regions?
K-Means Clustering • How to identify the dense regions for center selection? • Interactive selection via visualization When I see it, then I know it
K-Means Clustering • How to identify the dense regions for center selection? • Automatic selection via DB-Scan
K-Means Clustering • Inter-cluster overlapping
K-Means Clustering • Data Manifold Spectral Clustering
K-means Clustering • Partitional clustering approach • Each cluster is associated with a centroid (center point) • Each point is assigned to the cluster with the closest centroid • Number of clusters, K, must be specified • The basic algorithm is very simple Key Issues for K-Means: • K centroids • Similarity function • Objective function for optimization
K-means Clustering – Details • Initial centroids are often chosen randomly. • Clusters produced vary from one run to another. • The centroid is (typically) the mean of the points in the cluster. • ‘Closeness’ is measured by Euclidean distance, cosine similarity, correlation, etc. • K-means will converge for common similarity measures mentioned above. • Most of the convergence happens in the first few iterations. • Often the stopping condition is changed to ‘Until relatively few points change clusters’ • Complexity is O( n * K * I * d ) • n = number of points, K = number of clusters, I = number of iterations, d = number of attributes
Most common measure is Sum of Squared Error (SSE) For each point, the error is the distance to the nearest cluster To get SSE, we square these errors and sum them. x is a data point in cluster Ci and mi is the representative point for cluster Ci can show that micorresponds to the center (mean) of the cluster Given two clusters, we can choose the one with the smallest error One easy way to reduce SSE is to increase K, the number of clusters A good clustering with smaller K can have a lower SSE than a poor clustering with higher K Evaluating K-means Clusters
K-means Given a set of observations (x1, x2, …, xn), where each observation is a d-dimensional real vector, k-means clustering aims to partition the n observations into k (≤ n) sets S = {S1, S2, …, Sk} so as to minimize the within-cluster sum of squares (WCSS). In other words, its objective is to find: where μi is the mean of points in Si.
K-means Given an initial set of k means m1(1),…,mk(1) , the algorithm proceeds by alternating between two steps: Assignment step: Assign each observation to the cluster whose mean yields the least within-cluster sum of squares (WCSS). Since the sum of squares is the squared Euclidean distance, this is intuitively the "nearest" mean.
K-means Update step: Calculate the new means to be the centroids of the observations in the new clusters.
Optimal Clustering Sub-optimal Clustering Two different K-means Clustering Original Points
Problems with Selecting Initial Points • If there are K ‘real’ clusters then the chance of selecting one centroid from each cluster is small. • Chance is relatively small when K is large • If clusters are the same size, n, then • For example, if K = 10, then probability = 10!/1010 = 0.00036 • Sometimes the initial centroids will readjust themselves in ‘right’ way, and sometimes they don’t • Consider an example of five pairs of clusters
Solutions to Initial Centroid Problem • Multiple runs • Helps, but probability is not on your side • Sample and use hierarchical clustering to determine initial centroids • Select more than k initial centroids and then select among these initial centroids • Select most widely separated • Postprocessing • Bisecting K-means • Not as susceptible to initialization issues