Video Segmentation

1. Video Segmentation Brief on Labelling independently moving image regions

2. Motion Segmentation • Foreground(object) and Background(noise) Result could be a • Binary image, containing foreground only • Probability image, containing the likelihood of each pixel being foreground • Approaches • Motion-based (optical flow) • Color-based • Texture-based

3. Motion (Change) Detection • Static camera : changed and unchanged regions. • Moving camera : global and local motion regions. Limitations associated with motion estimation • Aperature problem : pixels in a flat image region may appear stationary even if they are moving as a result of an aperture problem (hence the need for hierarchical methods) • Occlusion Problem : erroneous labels may be assigned to pixels in covered or uncovered image regions as a result of an occlusion problem.

4. Motion Detection: using two frames : Image Differencing

5. Subtract Image • Compute pixel-wise • Subtract previous image from input image: • Usually the absolute distance is applied • Save image in last frame • Capture camera image • Subtract image • Threshold • Delete noise

6. 1 6 5 1 2 7 2 6 8 3 3 7 4 9 4 8 5 10

7. Threshold • Decide, when a pixel is supposed to be considered as a background pixel, or when it is to be considered as a foreground pixel: • Pixel is foreground pixel, if • Pixel is background pixel, if • Problem: What TH?!? • Save image in last frame • Capture camera image • Subtract image • Threshold • Delete noise

8. 5 1 2 6 3 7 4 8

9. 8 7 6 5 4 3 2 1

10. Dominant Motion : no optical flow • Spatio temporal intensity gradient. • Dominant motion segmentation • fitting a single parametric motion model • partition the frame in two pixel groups • Repeat step 1 only to well represented pixels group.

11. Image subtraction Thresholding Noise Removal Median Filter Bounding Box Centroid

12. Multiple Motion Segmentation • Multiple motion models compete against each other at each decision site. They consist estimating: • Motion within each region (motion model) • spatial support of each region • number of regions. • The problem : associate each pixel to the right motion model, while simultaneously estimating motions and supports. • Clustering (K-means, hough transform), • Maximum Likelihood (ML) (pixel based) and • Maximum APosteriori probability (MAP). • Region based label assignment

13. Optical flow estimation motion vectors at each frame Thresholding Morphological closing on the motion vectors

14. Object tracking using kalman filters (prediction)

15. Motion Model • Predicted position at time t: • Brownian Motion: According to a Gaussian model • 0’th order: • 1’th order: • Similar for y • 2’th order • Similar for y

16. Color Based segmentation : Background Subtraction • Use Neighborhood relation!! • Compare pixel with its neighbors!! • Weight them!! • Learn the background and its variations!! E.g. Gaussian models (mean,var) for each pixel!!! E.g. a Histogram for each Pixel • The more images you train on the better!! • Algorithm: • Consider each pixel (x,y) in the input image and check, how much it varies with respect to the mean and variance of the learned Gaussian models? • Calculate mean and variance for each pixel • Capture camera image • Subtract image (= motion) • Weight the distances (new) • Threshold according to variance • Delete noise

17. Color Segmentation with Histograms

18. Color Segmentation with Histograms brightness

19. Color Segmentation with Gaussian Distribution N(m, C)

22. Thank you