Introduction to Image Processing and Computer Vision

1. Introduction toImage Processing andComputer Vision Rahul Sukthankar Intel Research Laboratory at Pittsburgh and The Robotics Institute, Carnegie Mellon rahuls@cs.cmu.edu

2. Image Processing vs. Computer Vision • Image processing: • Image  image • e.g., de-noising, compression, edge detection • Computer vision: • Image  symbols • e.g., face recognition, object tracking • Most real-world applications combine techniques from both categories

3. Outline • Operations on a single image • Operations on an image sequence • Multiple cameras • Extracting semantics from images • Applications

4. Outline • Operations on a single image • Operations on an image sequence • Multiple cameras • Extracting semantics from images • Applications

5. What is an Image? • 2D array of pixels • Binary image (bitmap) • Pixels are bits • Grayscale image • Pixels are scalars • Typically 8 bits (0..255) • Color images • Pixels are vectors • Order can vary: RGB, BGR • Sometimes includes Alpha

6. What is an Image? • 2D array of pixels • Binary image (bitmap) • Pixels are bits • Grayscale image • Pixels are scalars • Typically 8 bits (0..255) • Color images • Pixels are vectors • Order can vary: RGB, BGR • Sometimes includes Alpha

7. What is an Image? • 2D array of pixels • Binary image (bitmap) • Pixels are bits • Grayscale image • Pixels are scalars • Typically 8 bits (0..255) • Color images • Pixels are vectors • Order can vary: RGB, BGR • Sometimes includes Alpha

8. What is an Image? • 2D array of pixels • Binary image (bitmap) • Pixels are bits • Grayscale image • Pixels are scalars • Typically 8 bits (0..255) • Color images • Pixels are vectors • Order can vary: RGB, BGR • Sometimes includes Alpha

9. What is an Image? • 2D array of pixels • Binary image (bitmap) • Pixels are bits • Grayscale image • Pixels are scalars • Typically 8 bits (0..255) • Color images • Pixels are vectors • Order can vary: RGB, BGR • Sometimes includes Alpha

10. Canny Edge Detector cvCanny(…) Images courtesy of OpenCV tutorial at CVPR-2001

11. Morphological Operations • Simple morphological operations on binary images: • erosion: any pixel with 0 neighbor becomes 0 • dilation: any pixel with 1 neighbor becomes 1 • Compound morphological operations:(composed of sequences of simple morphological ops) • opening • closing • morphological gradient • top hat • black hat • Aside: what is the “right” definition of “neighbor”?

12. Morphological Operations Dilatation IB Image I Erosion IB Opening IoB= (IB)B Closing I•B= (IB)B TopHat(I)= I - (IB) BlackHat(I)= (IB)-I Grad(I)= (IB)-(IB) Images courtesy of OpenCV tutorial at CVPR-2001

13. Hough Transform Goal: Finding straight lines in an edge image Canny edge + Hough xformcvHoughLines(…) Original image Images courtesy of OpenCV tutorial at CVPR-2001

14. Distance Transform • Distance for all non-feature points to closest feature point cvDistTransform(…) Images courtesy of OpenCV tutorial at CVPR-2001

15. Flood Filling cvFloodFill(…) grows from given seed point Images courtesy of OpenCV tutorial at CVPR-2001

16. Image Statistics • Statistics are used to summarize the pixel values in a region, typically before making a decision • Some statistics are computed over a single image: • Mean and standard deviation: cvAvg(…), cvAvgSdv(…) • Smallest and largest intensities: cvMinMaxLoc(…) • Moments: cvGetSpatialMoment(…), cvGetCentralMoment(…) • Others are computed over pairs/differences of images: • Distances/norms C, L1, L2: cvNorm(…), cvNormMask(…) • Others are computed over pairs/differences of images: • Histograms: • Multidimensional histograms: (many functions to create/manipulate) • Earth mover distance – compare histograms: cvCalcEMD(…)

17. Gaussian and Laplacian pyramids Image segmentation by pyramids Image Pyramids:Coarse to Fine Processing Images courtesy of OpenCV tutorial at CVPR-2001

18. Image Pyramids:Coarse to Fine Processing Original image Gaussian Laplacian Images courtesy of OpenCV tutorial at CVPR-2001

19. Pyramid-based Color Segmentation Images courtesy of OpenCV tutorial at CVPR-2001

20. Outline • Operations on a single image • Operations on an image sequence • Multiple cameras • Extracting semantics from images • Applications

21. Background Subtraction • Useful when camera is still and background is static or slowly-changing (e.g., many surveillance tasks) • Basic idea: subtract current image from reference image. Regions with large differences correspond to changes. • OpenCV supports several variants of image differencing: • Average • Standard deviation • Running average: cvRunningAvg(…) • Can follow up with connected components (segmentation): • could use “union find” or floodfill: cvFloodFill(…)

22. Optical Flow • Goal: recover apparent motion vectors between a pair of images -- usually in a video stream • Several optical flow algorithms are available: • Block matching technique: cvCalcOpticalFlowBM(…) • Horn & Schunck technique: cvCalcOpticalFlowHS(…) • Lucas & Kanade technique: cvCalcOpticalFlowLK(…) • Pyramidal LK algorithm: cvCalcOpticalFlowPyrLK(…)

23. Active Contours:Tracking by Energy Minimization • Snake energy: • Internal energy: • External energy: cvSnakeImage(…) Images courtesy of OpenCV tutorial at CVPR-2001

24. Real cameras exhibit radial & tangential distortion: causes problems for some algorithms. First, calibrate by showing a checkerboard at various orientations:cvFindChessBoardCornerGuesses() Then apply an undistorting warp to each image (don’t use a warped checkerboard!)cvUndistort(…) If the calibration is poor, the “undistorted” image may be worse than the original. Camera Calibration Images courtesy of OpenCV tutorial at CVPR-2001

25. Outline • Operations on a single image • Operations on an image sequence • Multiple cameras • Extracting semantics from images • Applications

26. Stereo Vision • Extract 3D geometry from multiple views • Points to consider: • feature- vs area-based • strong/weak calibration • processing constraints • No direct support in OpenCV, but building blocks for stereo are there.

27. View Morphing Images courtesy of OpenCV tutorial at CVPR-2001

28. Outline • Operations on a single image • Operations on an image sequence • Multiple cameras • Extracting semantics from images • Applications

29. Face Detection Images courtesy of Mike Jones & Paul Viola

30. Large Scale Painful! Small Scale Classical Face Detection Images courtesy of Mike Jones & Paul Viola

31. Viola/Jones Face Detector • Technical advantages: • Uses lots of very simple box features, enabling an efficient image representation • Scales features rather than source image • Cascaded classifier is very fast on non-faces • Practical benefits: • Very fast, compact footprint • You don’t have to implement it!(should be in latest version of OpenCV)

32. Principal Components Analysis High-dimensional data Lower-dimensional subspace cvCalcEigenObjects(…) Images courtesy of OpenCV tutorial at CVPR-2001

33. PCA for Object Recognition Images courtesy of OpenCV tutorial at CVPR-2001

34. PCA for Object Recognition Images courtesy of OpenCV tutorial at CVPR-2001

35. Outline • Operations on a single image • Operations on an image sequence • Multiple cameras • Extracting semantics from images • Applications

36. Shadow Elimination Idea: remove shadows from projected displays using multiple projectors OpenCV Techniques: Image differencing Image warping Convolution filters Matrix manipulation PosterCam Idea: put cameras in posters and identify who reads which poster OpenCV Techniques: Face detection Face recognition Unsupervised clustering Examples of Simple Vision Systems

37. Single Projector: Severe Shadows display screen P

38. Two Projectors: Shadows Muted display screen P-1 P-2

39. Dynamic Shadow Elimination display screen camera P-1 P-2

40. display screen camera P-1 P-2 Shadow Elimination: Challenges • Occlusion detection: what does a shadow look like? • Geometric issues: which projectors are occluded? • Photometric issues: how much light removes a shadow? • Performance: how can we do this in near real-time?

41. display screen camera P-1 P-2 Shadow Elimination: Solutions • Occlusion detection: difference image analysis • Geometric issues: single shadow-mask for all projectors! • Photometric issues: uncalibrated – feedback system • Performance: only modify texture map alpha values

42. Shadow Removal witha Single Mask

43. Shadow Elimination Algorithm Camera images Projected

44. PosterCam Overview • PosterCam Hardware: • Camera in each poster • Embedded computer in each poster (~ iPAQ) • Network connection to other posters

45. PosterCam Details • Face detection: Viola/Jones (no float ops) • Lighting compensation:histogram equalization • Pose variation:additional synthetic faces • Unsupervised clustering:k-means and nearest neighbor with non-standard distance metric

46. Tips on Image Processing and Coding with OpenCV • Use the OpenCV documentation only as a guide(it is inconsistent with the code) • Read cv.h before writing any code • OpenCV matrix functions work on images: e.g., cvSub(…) • Beware camera distortion: cvUnDistort(…) may help • Beware illumination changes: • disable auto gain control (AGC) in your camera if you are doing background subtraction • histogram equalization (often good for object recognition) • Image processing algorithms may require parameter tuning: collect data and tweak until you get good results

47. Digital Image ProcessingGonzalez & Woods,Addison-Wesley 2002 Computer VisionShapiro & Stockman,Prentice-Hall 2001 Computer Vision: A Modern ApproachForsyth & Ponce,Prentice-Hall 2002 Introductory Techniques for 3D Computer VisionTrucco & Verri,Prentice-Hall 1998 Reference Reading

48. The End Acknowledgments • Significant portions of this lecture were derived from the Intel OpenCV tutorial by Gary Bradski et al. at CVPR-2001 • Thanks to my former colleagues at Compaq/HP CRL for additional slides and suggestions: Tat-Jen Cham, Mike Jones, Vladimir Pavlovic, Jim Rehg, Gita Sukthankar, Nuno Vasconcelos, Paul Viola Contact rahuls@cs.cmu.edu if you need more information