Saliency או בולטות ויזואלית אם תרצו

1. Saliency או בולטות ויזואלית אם תרצו Presented to : Prof. Hagit Hel-Or Presented by: AvnerGidron

2. Saliency – definition • Saliency is defined as the most Prominent part • of the picture. In the last lecture Reemhas defined it as a part that takes at least one half of the pixels in the picture. We’ll see that it is not always the case, and Saliency has more than one definition.

3. Saliency – definition What is salient here?

4. Saliency – definition Answer:

5. Saliency – definition Here we can see that although the grass has more Variance in color and texture the horse is the salient part.

6. Saliency – definition Image can have more than one salient area, and As a result areas that are more salient than others: Also salient, but less. Salient areas:

7. Saliency – definition Our objective – saliency map:

8. Sometimes all you need are a few words of encouragement.

9. How would you divide this picture to segments?

10. A possible answer: Two segments: • The swimmer • The background

11. Motivation - application Image mosaicking: the salient details are preserved, with the use of smaller building blocks.

12. Motivation - application Painterly rendering – the fine details of the dominant objects are maintained, abstracting the background input Painterly rendering

13. So, what are we going to see today? • Explanation on Saliency in human eyes. • Automatic detecting single objects (Local). • Automatic detecting fixation points (Global). • Global + Local approach.

14. Saliency in human eyes

15. Saliency in human eyes Our eyes detect Saliency by: • First, the parallel, fast, but simple pre-attentive process, attracted to: • Movement. • High contrast . • Intensity. Will be attracted here

16. Saliency in human eyes Then, the serial, slow but complex attention process, that takes the points found in the first stage and chooses which one to focus on while detecting new information.

17. Saliency in human eyes Slow attention process – example: And then notice the cat and Baby. Firs focus here:

18. Saliency in human eyes Example for saliency map by eye tracking:

19. Detecting single objects One approach to saliency is to consider saliency as a single object prominent in the image An Algorithm using this approach is the Spectral Residual Approach

20. Spectral Residual Approach Try to remember from IP lessons. What did we say that image Consists of? That’s right!!! Frequencies

21. Spectral Residual Approach (1) Terns out, that if we will take the average frequency domain of many natural images, it will look like this:

22. Spectral Residual Approach (2) Based on this notion, if we take the average frequency domain and subtract it from a specific Image frequency domain we will get Spectral Residual

23. Spectral Residual Approach The log spec. 𝓁 of Image is defined in matlab as: ImageTransform= fft2(Image); logSpec= log(1+ abs(ImageTransform));

24. Spectral Residual Approach - example

25. Spectral Residual Approach will be defined as a blurring matrix sized :

26. Spectral Residual Approach Generally one takes average over many images to get the average spec but because we have only one image We can convolute it with to get an approximation. Then we can get:

27. Spectral Residual Approach At this stage, we’ll perform inverse fft and go back to The space domain. In matlab: SaliencyImage= ifft2(ImageSpecResidual);

28. Spectral Residual Approach And we will take a threshold to determine the Object map: The saliency map:

29. Detecting fixation points Another approach is to detect points in the image where the human eye would be fixated on. Not like spectral residual approach, which finds a single point, this approach may find more than one point. One algorithm that uses this approach is the one based on Information Maximization.

30. Information Maximization Before we start, let’s define a few things Self information: For a probabilistic event, with a probability of p(x), the self information is defined as:

31. Information Maximization An Attribute of self information is that the smaller the probability the larger the self information For example: But in self information:

32. Information Maximization Another thing we’ll explain is what does Independent Component Analysis (ICA) Algorithm. Given a random vector representing the data and a random vector representing the components, the task Is to transform the observed data , using a linear Static transformation as into maximally independent components .

33. Information Maximization ICA numeric example: We can see that is independent, and we would like to find .

34. Information Maximization The answer:

35. Information Maximization And in signals:

36. Information Maximization – ICA vs PCA PCA, Principal Components Analysis- a statistic method for finding a low dim. Representation for a large dimensional data. * Fourier basis are PCA components of natural images

37. Information Maximization – ICA vs PCA The different between them is that PCA find his Components one after the other, in a greedy way, finding the largest component each time, while paying attention to ortogonalty. the ICA works in parallel finding all the components at once, while paying attention to independency.

38. Information Maximization – ICA vs PCA PCA ICA

39. Information Maximization – max info algorithm We start with a collection of 360,000 Random patches and activate ICA on them, to get A which is a set of Basis Function.

40. Information Maximization – max info algorithm Now, we have the basis function that “created” the image, and we would like to know what are the coefficients of each basis function per pixel. We take the pseudoinverse of A, and multiply it with the image:

41. Information Maximization – max info algorithm The result of the unmixing is a set of coefficients. For pixel at location denote the i‘th coefficient , where his value is : In one dim:

42. Information Maximization – max info algorithm For each pixel at the location , we denote the probability that by . evaluates how “likely” the coefficient values at pixel are, compered to the neighboring pixel coefficients. We compute first the likelihood of each coefficient of separately.

43. Information Maximization – max info algorithm Similarity of the coefficients A little bit of math: distance of s,tto j,k. This Gaussian measures how “stable” are the coefficients where 𝛹 is pixel neighborhood, and describes the distance of s,tto j,k.

44. Information Maximization – max info algorithm We can see that for pixel j,k its coefficients are different from its surround. That’s Why is big and the prob. is low. On the contrary for pixel m,l, its coefficients are similar to The ones in its surrounding and that’s way this prob. Is high Pixel j,k Pixel m,l

45. Information Maximization – max info algorithm after computing the likelihood of each coefficient of separately, we denote – as:

46. Information Maximization – max info algorithm The more similar the pixel coefficients are to it’s neighbor‘s coefficients the lower the prob. And thus The smaller the self information, and vice versa.

47. Information Maximization For example in the follow image we can see that the white area will have little “stability” in the coefficients, and therefore small P(X) and so it will have large S.I. We can also notice that that fact go hand in hand with This area being prominent. Large self information

48. Information Maximization – max info algorithm Now, we can take the values of the self information and turn it in to a saliency map!!

49. And we get:

50. Information Maximization – max info algorithm And the results are: Information max. Human eye original