Physics-based Illuminant Color Estimation as an Image Semantics Clue

1. Physics-based Illuminant Color Estimation as an Image Semantics Clue November 8, 2009 Christian Riess Elli Angelopoulou Pattern Recognition Lab (Computer Science 5) University of Erlangen-Nuremberg

2. Illumination as an Image Semantics Clue Semantic analysis: Cues on objects, location, time Can we addilluminationcolor as a clue?

3. Existing Methods in Illuminant Estimation • Machine Learning-based, e.g. • Gamut Mapping (e.g. Forsyth) • Recent Gray Edge Variants (e.g. Gijsenij, Gevers) • Color by Correlation (e.g. Finlayson) • Physics-based, e.g. • Dichromatic Reflectance-based (e.g. Klinker et al.) • Intersection of diffuse color planes (e.g. Finlayson) • Inverse-Intensity Chromaticity (Tan et al.) Requirepropertraining Require“clean” input [1] [1] Tan, Nishino, Ikeuchi. “Color Constancy through Inverse-Intensity Chromaticity Space,” Journal of the Optical Society of America A, 21(3): 321-334, 2004.

4. Goal • Show that physics-based methods can be used • On arbitrary images, • Maybe sacrificing the exact estimate, • But giving higher level information (semantics).

5. The Inverse Intensity-Chromaticity Approach • Neutral Interface Assumption (NIA): specular color = illuminant • Image formation • Let • Chromaticity • Diffuse chromaticity • Specular chromaticity • Rewritten w/ chromaticities Diffuse geometry Diffuse reflectance Specular geometry Specular reflectance

6. The Inverse Intensity-Chromaticity Approach • Tan et al. showed , where • NIA: specular color = light color = . • The illuminant chromaticity relates linearly to the sum of intensities and the pixel chromaticities . • Specular regions form a trianglein inverse-intensity space,pointing to the illuminant color.

7. Towards Automated Scene Assessment • Tan et al. propose to estimate the illuminant chromaticity using the Hough transform along the y-axis. • Hypothesis: The histogram shape gives clues on the illuminant estimate quality, and the illumination environment in general. Hough Spacealong y-axis Hough Space, as ahistogram in its own right

8. Influence of the Specularity Segmentation • Experiments with different specularity segmentations • R, G, B estimatessurprisingly stable • Noise componentshows as a broaderpeak.

9. Influence of the Specularity Segmentation • Experiments with similar scenes • Illuminant color estimates not sufficiently reliable:Left image illuminant color estimate(0.378, 0.303, 0.318),Right image illuminant color estimate(0.339, 0.341, 0.319).

10. Towards automated Self-assessment • Is it possible to detect unmet constraints (e.g. poor specularity segmentation)? • Shape of the histogram peak • Fit Gaussian to peak • Fit triangle to peak • Optimization criteria • Intersection area • Sum of squared differences

11. Multiple illuminants mix in the histogram Observed peakmight be amixture of noisecomponents frommultiple lightsources. Decompositioninto ROIsshows themixingproperty. Further Challenges on the Histogram Shape

12. Barely specularities, high amount of noise Indoor illumination, reddish interreflections Red/blue channels: Peak can be “hidden” Further Challenges on the Histogram Shape

13. Summary • This is preliminary work. • Shape of the histogram can be used as a semantic source of information. • Method is insensitive to variations in specularity segmentation. • Further analysis of histogram shape: • Impact of binning • Noise vs. multiple illuminants • Learn histogram shapes? • Iterative simultaneous segmentation and illuminant estimation

14. Image Sources • http://www.flickr.com • http://www.walks.ru • Barnard et al.: A Comparison of Computational Color-Constancy Algorithms, IEEE TIP 11(9), 985-996, 2002. Thank you for your attention!