Computational Colour Vision

1. http://www.colourtech.org Computational Colour Vision Stephen Westland Centre for Colour Design Technology University of Leeds s.westland@leeds.ac.uk June 2005 Oxford Brookes University

2. Computational Colour Vision Introduce some basic concepts - the physical basis of colour Phenomenology of colour perception (the problem) Computational approaches to how colour vision works Computational and psychophysical studies of transparency perception

3. E(l)P(l) E(l) C(l) = E(l)P(l) The colour signal C(l) is the product at each wavelength of the power in the light source and the reflectance of the object P(l) The Physical Basis of Colour

4. Cone spectral sensitivity S ∫ L = E(l)P(l)FL(l)dl ∫ M = E(l)P(l)FM(l)dl M ∫ S = E(l)P(l)FS(l)dl L

5. ∫ L = E(l)P(l)FL(l)dl Cone Responses ∫ M = E(l)P(l)FM(l)dl ∫ S = E(l)P(l)FS(l)dl Each cone produces a univariant response L S M Colour perception stems from the comparative responses of the three cone responses Colour is a perception – ‘the rays are not coloured’

6. P Indoors (100 cd/m2) Outdoors (10,000 cd/m2) 0.01 1 100 0.99 99 9900 Colour Constancy Objects tend to retain their approximate daylight appearance when viewed under a wide range of different light sources The visual system is able to discount changes in the intensity or spectral composition of the illumination WHY? / HOW?

8. noon sunset

9. X

10. X

11. L1 / L1W = L2 / L2W M1 / M1W = M2 / M2W S1 / S1W = S2 / S2W e1 = De2 L1W/L2W M1W/M2W S1W/S2W 0 0 0 0 0 0 L1 M1 S1 L2 M2 S2 D = e1 = e2 = Computational Explanation ∫ ∫ L1 = E1(l)P(l)FL(l)dl L2 = E2(l)P(l)FL(l)dl ∫ ∫ M1 = E1(l)P(l)FM(l)dl M2 = E2(l)P(l)FM(l)dl ∫ ∫ S1 = E1(l)P(l)FS(l)dl S2 = E2(l)P(l)FS(l)dl

12. brightest pixel is white grey-world hypothesis Practical Use – Colour Correction Camera RGB values vary for a scene depending upon the light source colour correction In order to correct the images we need an estimate of the light source under which the original image was taken

13. ∫ L = E(l)P(l)FL(l)dl ∫ M = E(l)P(l)FM(l)dl ∫ S = E(l)P(l)FS(l)dl Colour Constancy Adaptation is too slow to explain colour constancy “Any visual system that achieves colour constancy is making use of the constraints in the statistics of surfaces and lights” – Maloney (1986) Is it possible for the visual system to recover the spectral reflectance factors of the surfaces in scenes from the cone responses?

14. Basis Functions P(l)S wiBi(l) Using a process such as SVD or PCA we can compute a set of basis functions Bi(l) such that each reflectance spectrum may be represented by a linear sum of basis functions - a linear model of low dimensionality. If we use n basis functions then each spectrum can be represented by just n scalars or weights.

15. 1 Basis Function Original 1 BF P(l) = w1B1(l)

16. 2 Basis Functions Original 1 BF 2 BF P(l) = w1B1(l) + w2B2(l)

17. 3 Basis Functions Original 1 BF 2 BF 3 BF P(l) = w1B1(l) + w2B2(l) + w3B3(l)

18. How many Basis Functions are Required? About 99% of the variance can be accounted for by a 3-D model (Maloney & Wandell, 1986) But what proportion of the variance do we need to account for? 6-9 basis functions are required

19. Simultaneous Contrast

22. original original covered by filter original with small filter

23. ei,1/ei,2 = e'i,1/e'i,2 (Foster) Ratio under second light source Ratio under first light source i = {L, M, S} Colour Constancy - spatial comparisons “For the qualities of lights and colours are perceived by the eye only by comparing them with one another” (Alhazen, 1025) “… object colour depends upon the ratios of light reflected from the various parts of the visual field rather than on the absolute amount of light reflected” (Marr)

24. L1 =k L2 L’1 =k L’2 Spatial Comparison of Cone Excitations Retinex – Land and McCann (1971) Foster and Nascimento (1994)

25. e’1 e’2 e1 e2 Transparency Perception e1/e2 = e’1/e’2 (Ripamonti and Westland, 2001)

26. What is transparency? An object is (physically) transparent if some proportion of the incident radiation that falls upon the object is able to pass through the object.

27. What is perceptual transparency? Perceptual transparency is the process ‘of seeing one object through another’ (Helmholz, 1867) Physical transparency is neither a necessary or sufficient condition for perceptual transparency (Metelli, 1974) Even in the complete absence of any physical transparency it is possible to experience perceptual transparency

28. Perceptual transparency

29. Research Questions What mechanisms could drive perceptual transparency? What are the chromatic conditions that cause transparency? Could transparency and colour constancy be linked?

30. Perceptual transparency

31. T(l) e'i,1 e'i,2 Transparency and Spatial Ratios ei,1 ei,2 ei,1/ei,2 = e'i,1/e'i,2

32. Experimental Computational analysis to investigate whether for physical transparency the cone ratios are preserved Psychophysical study to investigate whether the invariance of spatial ratios can predict chromatic conditions for perceptual transparency Psychophysical study to compare the performance of the ratio-invariance model when the number of surfaces is varied

33. P'(l) = P(l)[T(l)(1-b)2]2 (Wyszecki & Stiles, 1982) Physical Model of Transparency (1-b)b2P3T6 (1-b)PT2 (1-b)bP2T4 b T opaque surface P

34. 1. A pair of surfaces P1(l) and P2(l) were randomly selected 2. A filter was randomly selected (defined by a gaussian distribution) 3. The cone excitations were computed for the surfaces viewed directly (under D65) and through the filter P1(l) P2(l) s ei,1/ei,2 e'i,1/e'i,2 lm Monte Carlo Simulation 4. Steps 1-3 repeated 1000 times

35. e e i,1 i,2 e e ' ' i,1 i,2 Monte Carlo Results e'i,1/e'i,2 ei,1/ei,2

36. e e i,1 i,2 e e ' ' i,1 i,2 Monte Carlo Results The ratios are approximately invariant Invariance is slightly better for the S cones Invariance decreases as the spectral transmittance decreases

37. xB xA xQ xP g Convergence xA xB xQ xP xP= a xA+ (1-a) g xQ= a xB+(1- a) g Da Pos, 1989, D’Zmura et al., 1997

38. Psychophysical Stimuli I (a) convergent (deviation  0) (b) invariant (deviation = 0) deviationi = 1 - [ei,1/ei,2]/[ e'i,1/e'i,2]

39. 5 3 d' 1 -1 -3 0 0.1 0.2 0.3 0.4 0.5 0.6 LMS deviations L M S Log. (L) Log. (S) Log. (M) Psychophysical Results I d'<0 indicates subjects' preference for convergent filter; d'=0 no preference; d'>0 indicates subjects' preference for invariant filter

40. Psychophysical Stimuli II

41. Psychophysical Results II

42. Conclusions Computational and pyschophysical studies show that the invariance of cone-excitation ratios may be a useful cue driving transparency perception Colour constancy and transparency perception may be related. Could they result from similar mechanisms, perhaps even similar groups of neurones? There are still a mass of unsolved problems in computational colour vision including the relationship between cone excitations and the actual sensation of colour

43. xB xA xQ xP g xA xB xQ xP xP= a xA+ (1-a) g xQ= a xB+(1- a) g xP= a xA xQ= a xB Cone excitations are transformed by a a diagonal matrix whose diagonal elements are all equal

44. xA xP= b xA xQ= b xB xB xQ xP Cone excitations are transformed by a a diagonal matrix whose diagonal elements are not necessarily all equal The two models can be made to be the same if the convergence model has no additive component and if the invariance model has equal cone scaling