Image Sequence Based Particle Filter for Point Tracking

1. Journées Thématiques Filtrage Particulaire Image Sequence Based Particle Filter for Point Tracking Elise Arnaud - IRISA Etienne Mémin - IRISA Bruno Cernushi-Frias - Universitad de Buenos Aires

2. Introduction • Objective : Point tracking in computer vision :Reconstruction of a point trajectory along a given image sequence Particular framework with no a priori knowledge • Problem on which many high level tasks depend : motion estimation, 3D reconstruction, dynamic vision, etc. • Applications: robotics, medical imaging, meteorological imaging, surveillance, etc.

3. Presentation 1. Introduction 2. Related works on point tracking 3. Why an image sequence based filter ? 4. Image sequence based particle filter 5. Application to point tracking 6. Results and comparison 7. Conclusion 8. Perspectives

4. Related works on point tracking 2.1 Assumptions 2.2 Matching approaches 2.3 Differential approaches 2.4 Use of filtering methods

5. Assumptions Related works on point tracking (1) Motion hypotheses : drifting point, constant velocity, constant acceleration, periodic motion • too difficult to model without any a priori knowledge • too restrictive in the case of abrupt changes of the trajectory (2) Relative position conservation within a rigid geometric structure of the scene • Problem in the case of points moving independently of the scene (3) Luminance pattern conservation along the trajectory

6. Matching Approaches Related works on point tracking Luminance pattern conservation • Maximization of a similarity criterion between the target point and the candidate point • Similarity criteria based on a description of the luminance pattern • Necessity of exhaustive research  time consuming • Most similarity criteria are not invariant to affine changes • Comparative study of the most used criteria [Aschwanden92]

7. Differential approaches Related works on point tracking Luminance pattern conservation • Differential formulation of a similarity criterion • Point intensity conservation  optical flow constraint • Sum of square difference  Shi-Tomasi-Kanade tracker [Shi94]

8. Use of filtering methods Related work on point tracking • To design a tracker more robust to outliers and occlusions • State of the filter : feature position (+ intensity + velocity ) • Kalman filter for tracking in an image sequence : [Nguyen01] [Meyer94] [Ricquebourg00]

9. Why an image sequence based filter ? 3.1 Notations 3.2 Which type of available measures ? 3.3 Which type of dynamic ? 3.4 Objectives and assumptions

10. Notations Why an image sequence based filter ? state of the system at time k trajectory from time 0 to time k measure at time k measures from time 1 to time k random vector corresponding to an image at time k image sequence from time 0 to time k

11. Which type of available measures ? Why an image sequence based filter ? • All available information contained in the image sequence Ideal case: measure = image • Often impossible to specify the relationship between state and image ( too complex structure and large size ) Use of a condensed information obtained from the sequence • Highly nonlinear form with respect to the images • Simple form with respect to the state Measure equation :

12. Which type of dynamic ? Why an image sequence based filter ? • Dynamic a priori (constant velocity, periodic motion etc. ) • Dynamic captured from learning [Blake98] • If no a priori knowledge : Use of a dynamic obtained from the image sequence Dynamic equation :

13. Objectives and assumptions Why an image sequence based filter ? • Objective : estimation of the trajectory given all available data, i.e. measures and image sequence • Assumptions :

14. Objectives and assumptions Why an image sequence based filter ? • Dynamic equation and measures depend on the sequence • Such a dependency has to be taken into account • Conditioning with respect to the image sequence in the equations of the filter Definition of an image sequence based filter • Case of nonlinear dynamic or nonlinear measure equations Definition of an image sequence based particle filter

15. Image sequence based particle filter • Objective : approximation of • Knowledge of - N samples according to the importance function- N associated normalized weights • Non-normalized weights given by :

16. Image sequence based particle filter • Recursive equation of the importance function assumed • Recursive formulation of the weights • Increase over time of the weights’ variance • Optimal importance functionminimizes the weights’ variance conditioned upon • New recursive formulation of the weights

17. Image sequence based particle filter • Initialisation for k = 1 … n • Measures Evaluate the measure and the dynamic from the image sequence • Prediction • Weights update • Resampling • Trajectory estimation

18. Application to point tracking Proposed tracker combines a dynamic and some measures all depending on the image data State of the filter : point position 4.1 Conditional observation equation 4.2 Conditional dynamic equation 4.3 Image sequence based particle filter for point tracking

19. Conditional observation equation Application to point tracking • Restriction : linear observation equation • : result of an estimation process on the image sequence • : Gaussian white noise conditionally to is a Gaussian function

20. Conditional observation equation Application to point tracking • : most similar point to in image • Several matching criteria can be used to quantify the similarity • Criterion used invariant to affine transformations, originally defined for image matching applications [Schmid97] • Measure carries enough information to write :

21. ... ... Conditional observation equation Application to point tracking initial point initial vector of characteristics - characterization of the luminance pattern in the neighborhood - invariant to affine transformations selected measure  associated to the vector the most similar to the initial vector

22. Conditional dynamic equation Application to point tracking • Use of an instantaneous motion measure from image data • : Gaussian white noise conditionally to • : motion vector of • : motion parameters vector, result of an estimation process between images and

23. Conditional dynamic equation Application to point tracking • Point belonging to the background : - motion parameters vector corresponds to a unique global linear motion - linear dynamic equation Image sequence based Kalman filter • Point with a motion different from the global motion : - motion parameters vector corresponds to a local linear motion - nonlinear dynamic equation Image sequence based particle filter

24. Image sequence based particle filter for point tracking Application to point tracking and Gaussians • Knowledge of the optimal importance function, which is Gaussian. • Knowledge of the distribution involved in the weights recursion, which is also Gaussian

25. Position in image k Evaluate the observation by a matching method New image at time k New particles and associated weights Resampling For each particle, evaluate the motion vector by a differential method Image sequence based particle filter for point tracking Application to point tracking

26. Results and comparisons Sequence Caltra - 40 frames (190  180 pixels) Shi-Tomasi-Kanade tracker Image sequence based particle filter for tracking

27. Results and comparisons Sequence Meteo - 14 frames (256  512 pixels) Image sequence based particle filter for tracking

28. Conclusion • Definition of an image sequence based particle filter • Application to point tracking • Proposed tracker combines a dynamic and some measures all depending on the image data • No a priori knowledge • Trajectories undergoing abrupt changes • Sequence with a cluttered background

29. Perspectives • Consider a confidence measure of the observation • Include occlusion rules in the tracker • Test other dynamic calculated from image sequence : use of dense motion field • Application to fluid imagery ( meteorological sequence )