Visual Tracking

1. Visual Tracking CMPUT 615 Nilanjan Ray

2. What is Visual Tracking • Following objects through image sequences or videos • Sometimes we need to track a single object, sometimes a number of them • Sometimes we just track the object centroid, sometimes entire object boundary (shape)

3. Theoretical Foundation • Visual tracking is a “state” estimation problem • Bayesian inference is at the heart of visual tracking; it is called sequential Bayesian estimation • We form the posterior probability of the state, given all evidence or measurements up to the current time point • Inference is performed from the posterior density

4. Setting The Stage Some notations: Xt: unknown state we want to estimate at time point t; e.g., object centroid Zt: Measurement/observation made at time point t; e.g., image intensities The sequential estimation model assumes that we know three probability densities: p(X0): The initial state density p(Xt|Xt-1): State transition density or motion model p(Zt|Xt): Measurement/observation/likelihood density

5. Sequential Bayesian Estimation(AKA Sequential Bayesian Filtering) • We want to recursively estimate the state Xt given the observations Z1:t = {Z1, Z2, …, Zt}

6. Sequential Bayesian Estimation… Filter: Prediction Previous posterior Likelihood/observation density Bayes’ Rule: Current posterior

7. Bayes’ Rule Derivation Conditional probability rule Marginal density rule Also, because measurement Zt is conditionally independent on the current state Xt: So, we have the sequential Bayes’ rule:

8. Filter Derivation Rule of marginal density Rule of conditional probability Also, note that Xt is conditionally independent on Xt-1(Markovianity), so: Thus we have the filter rule:

9. Important Assumptions • Observation is conditionally independent on the current state • Current state is conditionally independent on the immediate previous state

10. Computation • Theory is all good, however we need to show people that it works in practice… • We will study Particle filter, the framework that can compute the recursive state estimation, i.e., sequential Bayesian estimation • We will also study Kalman filter, a popular sequential state estimation technique with some more assumptions

11. What is a Particle Filter? Let the particles represent the previous density So, the filter step is now: And the Bayes’ rule is now: We need to generate the current particle set from p(Xt|Zt): Particle filter

12. Factored Sampling Let h(x) = f(x)g(x) is a product of two functions, where say, g(x) is a density and f(x) is another non-negative function Factored sampling says that to represent h(x) non-parametrically by a set of particles, generate samples from g(x) and assign weights by f(x) i.e., {(s1, w1), …, (sn, wn)}, where si are generated from g(x) and wi = f(si). This is closely related to another sampling method called importance sampling.

13. Conditional Density Propagation (CONDENSATION) This a product of two functions: (1) and (2) Following the principle of factored sampling, CONDENSATION generates samples from (1) And assigns weights using (2)

14. Samples From a Mixture Density is a mixture density Notice that To generate samples from the mixture density these two steps are followed:

15. CONDENSATION Algorithm

16. How to estimate the state? • OK, we generated samples, what do we do with them: estimate the current state: h is any function of the state, for example when h(x) = x, we are performing state estimation

17. Other PFs • To date lots of particle filters have been proposed: • Sequential importance re-sampling (SIR) • Auxiliary particle filter (APF) • Likelihood particle filter • Rao-Blackwellized particle filter • A ton others • A leading researcher in PF : ArnoudDoucet

18. Some Points to Ponder about PF • The good point about PF is that it can handle very general likelihood and motion models • PF inherits a serious shortcoming from non-parametric density representation – curse of dimensionality – when the state space x is large, for example large multiple number of objects etc.