1 / 9

Optimization of MOPS Object Creation and Database Insertion for MPC Data Processing

This document outlines the processes involved in calculating and creating MOPS (Moving Object Processing System) objects from MPC (Minor Planet Center) data. It includes insights on the detection and insertion of orbits, tracklets, and derived objects within the database. Key performance metrics reveal transaction rates under various configurations, identifying methods to enhance efficiency, reduce detection counts, and optimize database interactions. The major findings contribute to understanding MOPS implementations, particularly regarding astronomical datasets like Spacewatch and NEOS model data.

donagh
Télécharger la présentation

Optimization of MOPS Object Creation and Database Insertion for MPC Data Processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MPC Data Ingest Notes

  2. Orbit calculation and creation of MOPS objects • Objects include Detections, Orbits (original MPC orbits), Orbits (JPL-calculated) • Stored to disk • ~19 h running time with one CPU

  3. Additional object creation and DB insertion via MOPS routines • Object creation includes Tracklets, DerivedObjects • 159366 Tracklets • 159357 DerivedObjects • DB insertion includes Detections, Tracklets, DerivedObjects • 31778048 Detections • 32096771 or ~32.09*10^6 objects

  4. DB transaction rates • Additional object creation and DB insertion via MOPS routines • 32.09*10^6 objects + 159357 records in table derivedobject_attrib = ~32.25*10^6 records • 36.5 h running time or ~245 transactions/sec

  5. DB transaction rates • Batch DB insert using optimized SQL directly on the database server • 32.25*10^6 records • 8.73 h running time • ~1025 transactions/sec • Batch DB insert using optimized SQL over the network • 9.75 h running time • ~918 transactions/sec

  6. MOPS overhead • Retrieving MOPS objects from disk and calling the insert method for each object • Includes various data integrity checks • Prepares SQL statements • Executes SQL statements • Processes database return values • 26.75 h additional processing time versus straight SQL insert

  7. Improving the situation • Reducing number of detections • Date-limiting observations • Optimizing the detections table definition • Removing indexes

  8. DB transaction rates • Batch DB insert using optimized SQL with the removal of six indexes in detections table • 32.25*10^6 records • 55 min running time • ~9772 transactions/sec • Undetermined how indexes affect performance using MOPS routines

  9. Conclusions • 245 transactions/sec is a realistic value for object insertion (mostly Detections) via MOPS • Scale of MPC data and/or my implementation may be problematic • Inserted Spacewatch fields and detections along with NEOS model SSM to MPC data instance • Normal runtime for MOPS simulation using Spacewatch data and NEOS model is reported to be half a night • Started DTCTL on Friday, still running… • Received two interruptions with error “Too many links” in attributeOrbits

More Related