Image Processing Techniques with Emgu.CV: Filter Convolution and Blurring

1. EEC-693/793Applied Computer Vision with Depth Cameras Lecture 20 Wenbing Zhao wenbing@ieee.org

2. Outline Image processing with Emgu CV Image Filtering Edge detection

3. Image Processing vs. Computer Vision • Image processing mainly deals with getting different representations of images by transforming them in various ways • Computer vision is concerned with extracting information from images so that one can make decisions

4. Image Filters • Image filter: a function that takes the original pixel and gives an output that is proportional in some way to the info contained in the original pixel • Image filters are 2 dimensional matrix, referred to as a kernel • Images are 2 dimensional • A filter matrix is a discretized version of a filter function • A filter for filtering out vertical edges: 0 0 0 0 0 0 0 0 0 0 -1 -2 6 -2 -1 0 0 0 0 0 0 0 0 0 0 // Filter kernel for filtering out vertical edges float vertical_fk[5][5] = {{0,0,0,0,0}, {0,0,0,0,0}, {-1,-2,6,-2,-1}, {0,0,0,0,0}, {0,0,0,0,0}}; // Filter kernel for filtering out horizontal edges float horizontal_fk[5][5] = {{0,0,-1,0,0}, {0,0,-2,0,0}, {0,0,6,0,0}, {0,0,-2,0,0}, {0,0,-1,0,0}};

5. Applying Filters (Convolution) • Element-wise multiplication between elements of the kernel and pixels of the image • A function is used to calculate a single number using the result of all these element-wise multiplications • This function can be sum, average, minimum, maximum, or something very complicated • The value thus calculated is known as the “response” of the image to the filter at that iteration • The kernel is shifted to the right and if necessary down

6. Convolving Filter: Example • From: http://www.roborealm.com/help/Convolution.php • A grayscale image (10x10)

7. Convolving Filter: Example • A filter for line detection • The row=2, col=2 pixel and its neighborhood • Multiply the filter values with the image block: work with each pixel and its 3x3 neighborhood

8. Convolving Filter: Example (-34)+(-22)+(-77)+(-50)+(1200)+(-77)+(-93)+(0)+(-77) = 770 • Sum of all values: • Divide by the divisor and add the bias • Divisor and bias might be needed to keep pixel value within 0 to 255 • Typically divisor=1, bias=0: final result remains to be 770 • If new pixel value is > 255, set it to 255 • If new pixel value is < 0, set it to 0 • Hence, new pixel value for row=2, col=2 is 255

9. Convolving Filter: Example • Continue with all other 3x3 blocks using original values • Next block

10. An Implementation of 2D Filtering private Image<Bgr, Byte> filterImage(Image<Bgr, Byte> image, double[,] filter, double factor, double bias) { Image<Bgr, Byte> filtered = new Image<Bgr, Byte>(image.Width, image.Height); int w = image.Width; int h = image.Height; int filterWidth = filter.GetLength(0); int filterHeight = filter.GetLength(1); byte[, ,] data = image.Data; byte[, ,] filteredData = filtered.Data; for (int y = 0; y < w; ++y) { for (int x = 0; x < h; ++x) { double red = 0.0, green = 0.0, blue = 0.0; for (int filterY = 0; filterY < filterWidth; filterY++) { for (int filterX = 0; filterX < filterHeight; filterX++) {

11. An Implementation of 2D Filtering { int imageY = (y - filterWidth / 2 + filterX + w) % w; int imageX = (x - filterHeight / 2 + filterY + h) % h; red += data[imageX, imageY, 2] * filter[filterX, filterY]; green += data[imageX, imageY, 1] * filter[filterX, filterY]; blue += data[imageX, imageY, 0] * filter[filterX, filterY]; } byte r = (byte)Math.Min(Math.Max((int)(factor * red + bias), 0), 255); byte g = (byte)Math.Min(Math.Max((int)(factor * green + bias), 0), 255); byte b = (byte)Math.Min(Math.Max((int)(factor * blue + bias), 0), 255); filteredData[x, y, 0] = b; filteredData[x, y, 1] = g; filteredData[x, y, 2] = r; } } } return filtered; }

12. Filtering Results

13. Blurring Images • Blurring an image is the first step to reducing the size of images without changing their appearance too much • An image can be thought of as having various “frequency components” along both of its axes’ directions • Edges have high frequencies, whereas slowly changing intensity values have low frequencies • Vertical edge creates high frequency components along the horizontal axis of the image and vice versa. • Finely textured regions have high frequencies (pixel intensity values in it change considerably in short pixel distances)

14. Blurring Images • Reduce the size of an image: remove high-frequency components from it • Smooth out those high-magnitude short-interval changes • Image Intensity: Sum of the values at a pixel • RGB color image • Three channels: R, G, B • Each color channel’s intensity is the value of the color • Low intensity: dark • High intensity: whiter

15. Blurring Images • Blurring can be accomplished by replacing each pixel in the image with some sort of average of the pixels in a region around it • To do this efficiently, the region is kept rectangular and symmetric around the pixel, and the image is convolved with a “normalized” kernel • Normalized because we want the average, not the sum

16. Blurring Images: Kernels 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 • Box kernel • Sum: 25 • Gaussian kernel • Sum: 256 1 4 6 4 1 4 16 24 16 4 6 24 36 24 6 4 16 24 16 4 1 4 6 4 1

17. Blurring Images: Results • Reuse the same implementation as that for image filtering except we must normalize the calculation • Divide by the sum of the kernel used

18. Edge Detection • Edges: points in the image where the gradient of the image is quite high • Gradient: change in the value of pixel intensity • The gradient of the image is calculated by calculating gradient in X and Y directions and then combining them using Pythagoras’ theorem • The angle of gradient can be calculated by taking the arctangent of the ratio of gradients in Y and X directions, respectively • Kernels for calculating gradients along x and y direction • X direction: y direction: -3 -10 -3 0 0 0 3 10 3 -3 0 3 -10 0 10 -3 0 3

19. Canny Edges • The Canny algorithm uses some postprocessing to clean the edge output and gives thin, sharp edges • Canny algorithm • Remove edge noise by convolving the image with a normalized Gaussian kernel • Computer x and y gradients by using kernels: • Find overall gradient and gradient angle. Angle is rounded off to 0, 45, 90 and 135 degrees • Non-maximum suppression: A pixel is considered to be on an edge only if its gradient magnitude is larger than that at its neighboring pixels in the gradient direction • Hysteresis thresholding: use two thresholds • A pixel is accepted as an edge if its gradient is higher than the upper threshold, rejected if smaller than lower threshold • For in-between pixels, it is accepted as an edge only if it is connected to a pixel that is an edge -1 -2 -1 0 0 0 1 2 1 -1 0 1 -2 0 2 -1 0 1

20. Canny Edge Detector API public Image<TColor, TDepth> Canny(TColor thresh, TColor threshLinking); • Defined for Image<> class • Parameters: • thresh: The threshhold to find initial segments of strong edges • threshLinking: The threshold used for edge Linking • Returns: The edges found by the Canny edge detector

21. Corners Detection • A corner can be defined as the intersection of two edges • A corner can also be defined as a point for which there are two dominant and different edge directions in a local neighborhood of the point

22. Building ImageFiltering App • Create a new C# WPF project named “ImageFiltering” • Add references, copy opencv dlls to path, configure project property for x64 bit target • Design GUI • Two image controls, 2 labels, a textbox, 6 buttons • Adding code

23. Building ImageFiltering App

24. Building ImageFiltering App • Import name spaces • using System.Diagnostics; • using System.Drawing; • using System.Text; • using Emgu.CV; • using Emgu.CV.Features2D; • using Emgu.CV.Structure; • using Emgu.Util; • using System.Runtime.InteropServices; // for DllImport • Add member variable • Image<Bgr, Byte> img;

25. Building ImageFiltering App • Core method for convolving (portion shown earlier) • Takes an Image<> and a filter (kernel) with parameters • For blurring, filter must be normalized • Returns the filtered image private Image<Bgr, Byte> filterImage(Image<Bgr, Byte> image, double[,] filter, double factor, double bias, bool normalize=false) if (normalize) { // calculate the sum of all elements in the kernel double sum = 0; for (int i = 0; i < filterWidth; i++) { for (int j = 0; j < filterHeight; j++) { sum += filter[i, j]; } } // need to normalize the filter kernel for (int i = 0; i < filterWidth; i++) { for (int j = 0; j < filterHeight; j++) { filter[i, j] = filter[i, j] / sum; } } }

26. Building ImageFiltering App • Filters used: vertical, horizontal, blur, sharper • Always use factor=1.0 and bias=0 // Filter kernel for detecting vertical edges double[,] verticalFilter = new double[,] { { 0, 0, 0, 0, 0 }, { 0, 0, 0, 0, 0 }, { -1, -2, 6, -2, -1 }, { 0, 0, 0, 0, 0 }, { 0, 0, 0, 0, 0 } }; // Filter kernel for detecting horizontal edges double[,] filter = new double[,] { { 0, 0, -1, 0, 0 }, { 0, 0, -2, 0, 0 }, { 0, 0, 6, 0, 0 }, { 0, 0, -2, 0, 0 }, { 0, 0, -1, 0, 0 } }; // Filter kernel for blurring (Gaussian kernel) double[,] filter = new double[,] { { 1, 4, 6, 4, 1 }, { 4, 16, 24, 16, 4 }, { 6, 24, 36, 24, 6 }, { 4, 16, 24, 16, 4 }, { 1, 4, 6, 4, 1 } }; // sharper filter double[,] filter = new double[,] { { -1, -1, -1 }, { -1, 9, -1 }, { -1, -1, -1 } };

27. Building ImageFiltering App • Edge detection code //Convert the image to grayscale and filter out the noise Image<Gray, Byte> gray = img.Convert<Gray, Byte>().PyrDown().PyrUp(); // to remove noise Gray cannyThreshold = new Gray(180); Gray cannyThresholdLinking = new Gray(120); Image<Gray, Byte> cannyEdges = gray.Canny(cannyThreshold, cannyThresholdLinking); image2.Source = ToBitmapSource(cannyEdges);

28. Building ImageFiltering App // Create OpenFileDialog Microsoft.Win32.OpenFileDialog dlg = new Microsoft.Win32.OpenFileDialog(); // Display OpenFileDialog by calling ShowDialog method Nullable<bool> result = dlg.ShowDialog(); // Get the selected file name and display in a TextBox if (result == true) { Image<Bgr, Byte> newimg; try { newimg = new Image<Bgr, byte>(dlg.FileName); this.img = newimg; image1.Source = ToBitmapSource(this.img); } catch { MessageBox.Show("Invalide file format"); return; } } • Browse and load a new image