Download
1 / 6
60 likes | 195 Vues
DISC-Finder: A distributed algorithm for identifying galaxy clusters. Two galaxies are “friends” if they are close to each other. We analyze an undirected graph, where galaxies are vertices and their “friendships” are edges. We need to identify its connected components.
E N D
DISC-Finder:A distributed algorithmfor identifying galaxy clusters
Two galaxies are “friends” if they are close to each other • We analyze an undirected graph, where galaxies are vertices and their “friendships” are edges • We need to identify its connected components Friends-of-Friends (FoF) technique: Identification of galaxy clusters Sequential algorithms • Exact: O((n∙ log n)1.5) • Approximate: O(n)
Divide the space into “slightly overlapping” cubes Load balancing: • Randomly select a subset of galaxies • Apply the kd-tree construction to build a balanced partition for the subset • Use it for the full set of galaxies • Distributed computation: Apply a sequential FoF algorithm to find the clusters within each cube • Use any sequential FoF • Allocate different cores to cubes • Identify cross-cube edges and merge the respective clusters • Apply the union-find algorithm to thegalaxies in the cube overlaps Distributed procedure
REDUCER MAPPER MAPPER MAPPER REDUCER REDUCER Distributed procedure localclusters galaxysets apply localsequential FoF divide the spaceinto cubes
Advantages • Scalable: We can apply it to massive datasets and use all available cores • Black-box use of a sequential FoF:We can utilize any FoF algorithm • Hadoop friendly: We have mapped all main operations into the Hadoop framework, which has resulted in very compact code (800 lines)
14,800 mlngalaxies 500 mlngalaxies Scalability Time (min) 240 SCALES TO LARGE DATA SETS EFFECTIVELY USES AVAILBLE CORES 60 1000 mlngalaxies 15 4 8 16 32 64 256 128 Number of cores
