Statistical Object Identification, Tracking, and Analysis

1. Statistical Object Identification, Tracking, and Analysis Michael Turmon Jet Propulsion Laboratory/Caltech AISR Program Meeting NASA Ames Conference Center 4 April 2005

2. Object Tracking and Analysis: Overview Allow scientists in domains like solar physics and atmosphere & ocean circulation to understand great volumes of temporal data in directly informative terms • Identification: Find the objects in multispectral science images • Tracking: Link identified objects in series of images • Trajectory Analysis: Model and classify object tracks Identification Tracking Trajectory analysis Sunspot and Facula Regions in a Solar Quadrant 15 November 1998 and the next five days; using MDI imagery Move scientists beyond looking at pixels to understanding phenomena

3. Project Activities • Scope: Demonstrate object analysis technology in three application areas: solar physics, geophysics (GPS), oceans/atmospheres • Highlights (June 2001 – June 2004) • Sunspots tracked over seven years’ images • 100 GB of imagery distilled into 2500 object histories, < 1GB • Related high-cadence track dataset also analyzed • Integration of object tracks with DS9 browser • Fast Kalman models developed and tested for GPS time series • New hidden Markov classification of seismograph time series • Developed and used new constrained optimization methods • Future activities • Perform analysis of high-cadence solar data • Tune segmentation and object tracking models for HMI

4. Technology Overview • Identification • Per-class mixture models drive Markov random field segmentation • Trained using combination of expert-provided labels and unclassified pixels • Aggregate connected components into objects • Tracking • Compute index of object overlap (past -> future) • Associate current objects to past objects to optimize total overlap • Object analysis • Continuous and discrete modeling of object path and characteristics • Basic subroutines are Kalman smoother and forward-backward recursion

5. Light Intensity Magnetic Field Labeling by inferred statistical model Magnetogram Labeling Photogram Identification: Integrating Multimode Imagery Q • Can not distinguish classes from just one observable • Move beyond ad hoc threshold rules to allow arbitrary class separators • Select model by using sample images labeled by scientists F Flexible, general methods using statistical models to identify objects in images S 1: Experts identify classes in sample images Key: S(pot) F(acula) Q(uiet sun) Q S Q F N S 2: Learned model performs classification automatically

6. Identification: Partly-Labeled Data • Hand Labeling: time-consuming, expensive, asks much of scientists. • Data from some feature classes (e.g., background) is easy to identify; small amounts of labeled data can be obtained with care. • E.g., scatter plot at left: 15K quiet examples + 607 sunspot + 340 facula • Technical challenge: ensure atypical distribution of labeled data does not affect learned class proportions. • Developed methods using partly-classified data to bootstrap vast amounts of unlabeled data, seamlessly in same clustering algorithm. • Selected 100K examples from 10B total, 30K labeled — mostly quiet background. • Yields >20% improvement in sunspot classification accuracy, and >25% improvement in facula classification accuracy. Labeled Data Previous Feature->Class Map New Feature->Class Map facula Unlabeled Data quiet facula quiet sunspot sunspot

7. Identification: Results Turmon et al., “Statistical Pattern Recognition for Labeling Solar Active Regions: Application to SoHO/MDI Imagery,” Astrophysical Journal, March 20 2002, 396-407.

8. Feature Identification: Publications • The mixture modeling work appeared in: • Mixtures-2001, “Recent Developments in Mixture Modelling,” Hamburg • Compstat-2004, Prague, as “Symmetric Normal Mixtures” • Work comparing our MDI labelings to other observatories: • Harry Jones (Kitt Peak Nat’l Solar Obs.) & Steve Walton (San Fernando Obs.) • J. Pap, H. Jones, M. Turmon & L. Floyd, “Study of the SOHO/VIRGO Irradiance Variations using MDI and Kitt Peak images,” Proc. SOHO-11 Workshop, Davos, 2002. • H.P. Jones, M. Turmon, et al. “A comparison of feature classification methods for modeling solar irradiance variation,” 34th COSPAR Scientific Assembly, 2002. • Laslo Gyorfi at Debrecen Observatory, Hungary • L. Gyori, T. Baranyi, M. Turmon & J.M. Pap, "Comparison of image-processing methods to extract sunspots,” Proc. SOHO-11 Workshop, Davos, 2002. • L. Gyori, T. Baranyi, M. Turmon & J.M. Pap, “Study of differences between sunspot area data determined from ground-based and space-borne observations,” Adv. Space Res., April 2004. • Connections with irradiance • J. M. Pap, M. Turmon et al., “Magnetic Field & Long-Term Solar Irradiance Variations Over Solar Cycles 21 to 23,” AGU, 2003, San Francisco (poster).

9. Feature Identification: Infusion • This software will be used in the HMI data pipeline at Stanford • HMI imager will fly on board SDO, the first LWS mission • http://sdo.gsfc.nasa.gov and http://hmi.stanford.edu • HMI’s data volume is unprecedented in Solar Physics • 4096x4096 pixel images every 90 seconds • These data volumes make it more important to focus attention • HMI/SDO is the successor to MDI/SoHO, on which these results are based • The software has also been baselined by CNES Picard • Picard has been given the go-ahead by CNES for 2007 launch • http://smsc.cnes.fr/PICARD/ • Picard will measure tiny variations in solar diameter and shape • Active region recognition and rejection is important to its delicate results • Software must clear ITAR and licensing hurdles

10. Object Tracking Methods After Before • Associate objects in beforeand after images • Correlation-based tracker • Motion model: deterministic drift plus stochastic uncertainty • For sunspots or cyclones, have motion and correlation on the sphere • Correlation measure between a in A and b in B is D(a,b) • Solve assignment problem to match A up to A’: with P a permutation matrix Solution by linear programming • For our applications, key is to get deterministic drift correct B A

11. Object Tracking: Example Labeling Magnetogram

12. Object Tracking: Sunspots over seven years • Coordinates, size, intensity of 2500 sunspots from 1996-2003 • Over 100-fold reduction in data volume • June through July 1999 shown above • Ordered by central meridian passage (CMP) time • Successive sightings overplotted; extent indicated by bounding box • Enable quantitative studies of spot taxonomy Latitude Time (CMP) ––> Zoom view

13. Object Tracking: High-rate data • We also have tracks from four, three-month periods of high-cadence data from SOHO/MDI • Continuous telemetry gives full-disk images every minute • Unprecedented temporal resolution: 1500 images/day or 3GB/day • Small features we identify and track are tracers for motion of plasma in photosphere • High-cadence data give more samples for each region of interest

14. Object Analysis Methods: General • Premise: Learn from objects by modeling their evolution as noisy differential equations of several related types • Hidden Markov models: Finite-state machine controls time series • Divide behaviors into classes according to hidden discrete state u(t) • Kalman filters: Continuous-state machine controls time series • Explain or model behavior by hidden state vector u(t) • Track clustering: Discrete variable C selects object type • Extends clustering (the most usefulbaseline discovery algorithm) to thetemporal domain • All methods in this family generalizethe basic HMM/Kalman model • Crucially: subroutine re-use • Kalman smoother and forward-backward

15. Object Tracking: Ocean Eddies • Same technology tracks eddies in shallow-water ocean simulation (Toshio M. Chin, JPL) • State (position, size) of two labeled eddies through time, above left • Two subclasses of eddies are apparent, above right