Advances in Adaptive Weight Support Windows

1. Advances in Adaptive Weight Support Windows Barry McCullagh, Keimyung University Daegu

2. Presentation Overview • Advances in adaptive support weight windows: • Introduce the problems of fixed support weights. • Seminal paper by Yoon and Kweon. • Advances in the past 12-18 months. • Future research. • Questions and answers.

3. Correlation in stereovision • Correlation windows (support windows) are used to locate matching pixels in stereo image pairs.

4. Window comparison • Traditionally each pixel was given the same weight. • Comparison performed using SAD, SSD, NCC or other metric. • Different objects in a window?

5. Problems

6. Adaptive weights • First introduced by Yoon and Kweon from KAIST. • Importance of pixel depends on: • Color similarity to center pixel. • Distance to the center pixel.

7. Color similarity • Pixels in the blue boundary are more important than the other pixels because they are of similar color.

8. Euclidean Distance

9. Examples

10. Since Yoon and Kweon • Use of adaptive weights has become popular: • Geodesic support weights • Biologically inspired weighting • Disparity Calibration Systems

11. Geodesic Support Weights • Local Stereo Matching Using Geodesic Support Weights. • Honsi, Bleyer, Gelautz, Rhemann, ICIP 2009. • Looks at the path to the center pixel. • Support weight is high if the pixels on the path are of similar color. • Support weight is low if the pixels on the path are dissimilar.

12. Path to the center • Problem identified in this paper: • Similar colors separated by dissimilar colors might not belong to the same object. • Not at the same depth.

13. Original paper: some areas are given incorrectly high levels of support.

14. Geodesic Technique • Geodesic technique looks at the path from a pixel in the window to the center pixel. • Consistent colors indicate the same object and therefore the same disparity.

15. Results • Image • Original Weighting • Geodesic Weighting

16. Biologically Inspired Weighting • “Biologically and psychophysically inspired adaptive support weights algorithm for stereo correspondence” Lazaros Nalpantidis and Antonios Gasteratos • Desire to simulate human matching in computers. • Identify techniques humans use and model these in stereo matching programs.

17. Biological Matching • Color Matching • Circular Windows • Gestalt Laws • Psychophysically-based weight assignment. • All expanding on the Absolute Difference calculated between individual pixels at different disparities.

18. Color Matching – color space • Color images produce more accurate results than color. • Different color spaces produce different levels of accuracy: • CIELab better than RGB • However: • color sensors are RGB. • RGB computations are less demanding.

19. Color Matching – weighting factors • Human Visual System Weightings: • 0.299 Red • 0.587 Green • 0.114 Blue • Stereovision: Equal weightings • Equal amount of information.

20. Circular Windows • Most approaches use rectangular windows. • Biological model is better approximated using circular windows. • Eye is circular • Contribution of neighboring pixels is perfectly isotropic.

21. Gestalt Laws • Gestalt – relationships that bond single items to make a group. • Makes a pattern instead of parts. • Pattern has different characteristics than the parts. • These laws are useful to help locate matching objects in stereovision.

22. Gestalt Laws • Elements (pixels) making up a group (belong to a larger object) are governed by the following rules: • Proximity: elements that are close to each other. • Similarity: elements similar in an attribute (color, etc). • Continuity: elements that could belong to a smooth larger feature. • Common fate: elements that exhibit similar behavior. • Closure: elements that could provide closed curves. • Parallelism: elements that seem to be parallel. • Symmetry: elements that exhibit a larger symmetry.

23. Gestalt Laws for image processing • Three most basic laws can be used to assist matching: • Proximity (or equivalently Distance). • Intensity similarity (or equivalently Intensity dissimilarity). • Continuity (or equivalently discontinuity).

24. Psychophysically-based weight assignment • Assign support weights based on the human response. • Weber-Fechner law shows the relationship between perceived change in stimulus and actual change in stimulus

25. Combining these laws • Three basic Gestalt laws are combined: • Wtotal = Wdist * Wdisc * Wdissim

26. Performance • Less accurate than the original algorithm from Yoon and Kweon. • Provides a basis for modeling the human visual system in computing.

27. Expanding on Adaptive Weights • “Local stereo matching with adaptive support-weight, rank transform and disparity calibration” Zheng Gu, Xianyu Su, Yuankun Liu and Qican Zhang. • Uses adaptive weight as the starting point and applies additional techniques to the result.

28. Additional Steps • Rank Transform • Pixels assigned discrete weights based on similarity to center pixel. • Disparity Calibration • Selection of window • Disparity calculation

29. Rank Transform • Calculates the intensity differences between the center pixel and pixels in the support windows. • If corresponding pixels in the two support windows have different differences, then those pixels are not included

30. Disparity Calibration • Selection of window: applies adaptive weights but with stronger emphasis on color similarity. • Recalculation of disparity values based on those of surrounding pixels

31. Disparity recalculation • Examine the disparity values for all pixels in the support window. • Assign the most common disparity to that of the center pixel. • Removes outliers • Increase smoothness of the disparity map.

32. Performance • Performs better than the original adaptive weight and better than most local methods.

33. Future Research • Two main goals: • improve speed by • re-implementing algorithms on parallel architectures. • modifying algorithms to reduce the computational cost. • improve accuracy: • using more complex algorithms. • merging components of different algorithms.

34. Parallel Architectures • GPUs, CBE, Multi-core CPU • Powerful, ubiquitous and cost effective. • Efficient implementation of adaptive support weight algorithms will achieve real-time rates. • Adaptive Support Weight • implementable on parallel devices. • may be restricted by local memory, available instructions and order of execution.

35. Modifying Algorithms • Computation cost is proportional to window size: • sliding windows cannot be used in most cases. • ASW often use windows of 35x35 pixels or larger. • Investigate techniques to reduce the window size while maintaining accuracy: • hierarchical approaches. • adaptively sized windows.

36. Modifying Algorithms • Combining and modeling more of the Gestalt laws. • Varying the importance of these laws. • Geodesic supports show color is very important. • Are other laws more important?

37. Merging Algorithms • Adaptive Support Weights forms the basis of many algorithms. • e.g. ASW combined with rank transform, modified to allow use with curvelets. • What other techniques can ASW be combined with? • As an initial step. • As a final step.

38. Questions and Answers