1 / 8

Enhancing Scientific Data Access with the Terabyte Analysis Machine and Distance Machine

The Terabyte Analysis Machine (TAM) is designed for efficient parallel access to large scientific databases. Our initial focus is on the Distance Machine, which uses re-indexing and re-partitioning techniques for optimized nearest neighbor searches within the SDSS database. The Distance Machine framework operates across approximately ten nodes and employs a batch processing system for database management. Performance tests indicate that clustering techniques significantly boost analysis, making TAM a versatile tool for both astronomical and non-astronomical data studies.

brant
Télécharger la présentation

Enhancing Scientific Data Access with the Terabyte Analysis Machine and Distance Machine

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Introduction • The Cluster Environment • The Distance Machine • Scales • Performance Tests • Conclusions • Links

  2. Introduction The Terabyte Analysis Machine is a cluster designed to allow research into the use of large scientific databases. Our initial program is to explore efficient parallel database access (KDI) on large data set (ALDAP), via re-indexing and re-partitioning techniques, into a “griddified” environment.

  3. The Cluster Environment The Distance Machine will manage in one year ~10 nodes using Condor as a batch system. Possible SX database transfers will be managed via GDMP The client-server applications of the Distance Machine framework communicate via CORBA interfaces.

  4. The Distance Machine As an example of a database re-indexing and re-partitioning in the TAM framework, the Distance Machine is an implementation of an analysis engine for the kth nearest neighbor search. The Distance Machine uses the SDSS (SX) database (John Hopkins University) It extracts the relevant data from SX using theTag Extractor utility developed by Koen Holtman (Caltech)

  5. Scales • The SDSS database will have: • 15 Tbytes image database • 500 Gbytes catalog database • 250 million objects • 100 million galaxies (example of the search for cluster of galaxies) • 2.5 Kbytes object size • 500 byte tag object size (example of tag with about 50 parameters of interest) • 50 Gbytes tag database • Moving the tag database: • 50 Gbytes at 10 Mbytes/sec => 1.5 hours to transfer over network • 50 Gbytes at 30 Mbytes/sec => 0.5 hours to write to disk • Dimensionality of the problem: • 120 parameters (total) per object • 5 parameters used for cluster finding: ra, dec, g-r, r-i, and i'

  6. Performance Tests The Distance Machine finds Nearest Neighbors using the kd-tree technique, accessing directly the database. This technique is O(N logN) Re-clustering techniques improve performance

  7. Conclusions The Distance Machine implements a framework to study how re-indexing and re-partitioning techniques increase performance on parallel access to large data sets. It is a core engine for cluster finding analysis on the SDSS data. The Distance Machine can be used as a test bed for the GriPhyN tools (virtual NN data sets). It’s flexible enough to be used on data other than astronomy (e.g. “out of the box”: study of Nearest Neighbors techniques for particle tagging)

  8. Links • http://projects.fnal.gov/act/kdi.html • http://sdss.fnal.gov:8000/~annis/astro Compute.html

More Related