A Novel Method for Generation of Motion Saliency

1. A Novel Method for Generation of Motion Saliency Yang Xia, Ruimin Hu, Zhenkun Huang, and Yin Su ICIP 2010

2. Outline • Introduction • Itti’s model • Proposed Visual Saliency • Generation of motion feature map • Enhancement of motion sub-saliency map • Experiment Results • Conclusion

3. Introduction • Visual saliency • Bottom up saliency • Top down saliency • Applications • Image segmentation, motion detection, image/video compression…… • Motion saliency • Motion object is more salient to human vision system(HVS) than spatial contrast in video.

4. Itti’s Model • For Image • Spatial features • Color, Intensity, and Orientation • Feature maps • Combining the normalized activation maps • Visual Saliency Model For Video • Temporal features • Flicker and Motion

5. Itti’s Model • Motion feature map • Computed by spatially-shifted differences between Gabor pyramids from the current frame n and previous frame n-1 • Motion feature • The minimum captured object velocity at scale • : motion feature map for scale and orientation • : Gabor pyramid of original frame n • : the shifted Gabor pyramid of original frame n dx, dy: horizontal and vertical shift distance f : the frame rate

6. Itti’s Model • Drawback • Inaccurate when the objects move slowly • when the velocity is smaller than , none of the pyramidal scales can capture the movement • group into the background • Only the edge of object is labeled salient • Using spatially-shifted differences Multi reference frame Enhance motion saliency map by spatial saliency information

7. Proposed Visual Saliency • Generation of motion feature map • Multi reference frames to enhance the ability to capture object movement • Motion feature map • Processed by graph theory to form the activation map[1] (Reference frame) p n add two velocity profiles about and [1] J. Harel, C. Koch, and P. Perona, “graph-based visual saliency,” in Advances in Neural Information Processing Systems 19, Cambridge, MA: MIT Press, 2007

8. Proposed Visual Saliency • Enhancement of motion sub-saliency map • Spatial sub-saliency map • Find the point which belongs to salient object • Check un-salient point is near a salient point which has large saliency value both in motion and spatial sub-saliency maps, and its spatial saliency value is close to that of the salient point. • Update the motion saliency • Generate the whole saliency map

9. Enhancement of Motion Sub-saliency Map Top 25% of locations which have larger saliency values in SMS Top 5% of locations which have larger saliency values in SMM If and the difference of the spatial saliency values between and New saliency location set Motion saliency points Whole saliency map +

10. Experimental Results • Dataset in CAVIAR—ThreePastShop1cor • ROC(Relative Operating Characteristic) score between estimated saliency maps(ESMs) and ground-truth saliency maps(GSMs) Anchor1: Itti’s model Anchor2: Itti’s model using activation operator based on graph theory SMRF: saliency model with the multi-reference frames SMRF+STE: plus spatio-temporal enhancement

11. Experimental Results anchor1 anchor2 motion channel SMRF SMRF+STE anchor1 five channel anchor2 SMRF+STE

12. Conclusion • First analyze the drawback of Itti’s motion saliency model. • Propose a novel motion saliency modelin which motion saliency map is obtained through the multi reference frames, and enhanced by spatial saliency information.