CSSE463: Image Recognition Day 30

1. CSSE463: Image Recognition Day 30 • This week • Today: motion vectors and tracking • Friday: Project workday. First status report due. Please bring short printed copy to class: • what you did • what you didn’t and the technical challenges • what you will do next week. • Questions?

2. What is image flow? • Notice that we can take partial derivatives with respect to x, y, and time.

3. Image flow equations • Goal: to find where each pixel in frame t moves in frame t+Dt • E.g. for 2 adjacent frames, Dt = 1 • That is, Dx, Dy are unknown • Assume: • Illumination of object doesn’t change • Distances of object from camera or lighting don’t change • Each small intensity neighborhood can be observed in consecutive frames: f(x,y,t)f(x+Dx, y+Dy, t+Dt) for some Dx, Dy (the correct motion vector). • Compute a Taylor-series expansion around a point in (x,y,t) coordinates. • Gives edge gradient and temporal gradient • Solve for (Dx, Dy) • See answers to first quiz question now

4. Limitations • Assumptions don’t always hold in real-world images. • Doesn’t give a unique solution for flow • Sometimes motion is ambiguous • Look at last question on last quiz. • “Live demo” • http://www.cs.mcgill.ca/~chundt/coherence/

5. Aperture problem • When we only see a small part of the image, motion can be ambiguous • Lesson: By constraining ourselves to a small neighborhood, we lose the “big picture” • Solutions: • Assume pixels belong to same object and have same motion • Propagate constraints between pixels • Can still take many frames to converge • Doesn‘t work at occlusions (hidden parts of object) • Sensitive to outliers • Only track motion of “interesting points” Q1

6. Interest point tracking • Idea: corners and other easily-identifiable points can be tracked, and their motion is unambiguous. • Technique for calculating a set of interest points: • Loop over image: • Calculate an interest score for a small neighborhood around each pixel • If the score is above threshold, then add it to a point set to be returned. Q2

7. Stretch break • When is your group willing to present? Q3,4

8. Interest scores • One solution: • Take minimum variance of those found in 4 directions: • (3x3 shown, but would use bigger window) • Another related texture-based operator is based on directions with significant edge gradient. • Could also detect corners using Kirsch operators (similar to other edge operators).

9. Tracking interest points • Search for them directly using a template. • Choose one with highest correlation • Just need to search in small neighborhood if frame rate is high enough • Shapiro’s Alg. 9.3

10. Review: Correlation • Just the dot product between the template and a neighborhood in the image. • Idea: high correlation when the template matches. • Problem: always high correlation when matching with bright region • Solution: Normalize each region by subtracting mean before taking dot product

11. Matlab libraries exist to extract frames (still images) from video for processing.

12. Computing trajectories • Problem: what if unlabeled moving points belong to multiple trajectories? • Idea: want to maximize smoothness, defined as change in direction and change of speed. • Given two points pt, pt+1, thought to be related, calculate vt = pt+1 – pt. Then: Average speed compared to geom. mean direction

13. Computing trajectories • Then the “Greedy-Exchange” algorithm is used to choose which points belong to which trajectory. • Operates by creating trajectories and exchanging points if it would increase total smoothness. • Local optimum.

14. Example: Tracking Pedestrians with Human Models • http://www.umiacs.umd.edu/users/hismail/Ghost_outline.htm • Thanks to Thomas Root for link

15. Another approach • The Kalman filter is a probabilistic model that combines noisy measurements with the expected trajectory of the object. It works even with occlusion. • See http://www.cs.unc.edu/~welch/kalman/ for a start.