Variational Image Restoration

1. Variational Image Restoration Leah Bar PhD. thesis supervised by: Prof. Nahum Kiryati and Dr. Nir Sochen* School of Electrical Engineering *Department of Applied Mathematics Tel-Aviv University, ISRAEL

2. What is image Restoration? • Image is degraded by deterministic (blur) and random (noise) processes. • Blur is assumed as linear shift invariant process with additive noise. • Camera out of focus • Motion blur • Atmospheric turbulence • Sensor noise • Quantization • Inverse problem which has been investigated for more than 40 years. • Given the image g and the blur kernel h, restore the original image f .

3. Image Restoration - Applications Microscopy

4. Image Restoration - Applications Astronomy

5. Image Restoration - Applications Medical Imaging

6. Bayesian and Variational Viewpoints Spatial domain: Frequency domain: Assuming Gaussian distribution of the noise • Maximum Likelihood • Variational • Ill-Posed Solution Noise amplification In high frequencies • Pseudo inverse Filter

7. Bayesian and Variational Regularization • Maximum a posteriori prob. MAP smoothness prior • Variational (Tikhonov , 1977) • Solution – Wiener Filter (over smoothing)

8. Edge Preservation observed image - g recovered image - f Image Deconvolution Image Denoising Preserve Edges Edges are very important features in image processing, and therefore have to be preserved.

9. Total Variation Regularization Wiener Total variation Rudin, Osher, Fetami 1992

10. Mumford-Shah Segmentation (Mumford and Shah, 1985) data fidelity gradients within segments total edge length Canny edges M-S edges Original Image is modeled as piecewise smooth function separated by edges Ω: image domainK: edge setf: recovered imageg: observed image

11. Deconvolution with Mumford-Shah Regularization data fidelity gradients within segments total edge length L. Bar, N. Sochen, N. Kiryati, ECCV 2004 M-S functional: difficult to minimize (free-discontinuity problem). Solution is via the G-convergence framework (Ambrosio and Tortorelli 1990) Strategy: approximate the solution by approximation of the problem

12. G-convergence • A sequence G-converges to if: • liminf inequality • existence of recovery sequence Example: Fj(u)=sin (ju) uj=1.5p n/j (De Giorgi, 1979) G-lim(Fj)=-1

13. G-convergence Fundamental theorem of G-convergence: Suppose that and let a compact set exist such that for all j, then . Moreover if uj is a converging sequence such that then its limit is a minimum point for F.

14. * Let satisfy Proof: There exists a subsequence converging to some u, such that This is satisfied for every u and in particular

15. Deconvolution with Mumford-Shah Regularization data fidelity gradients within segments total edge length v(x): smooth function v(x)~0 at edges v(x)~1 otherwise (in segments)

16. Deconvolution with Mumford-Shah Regularization • Iterate • Minimize with respect to v by Euler equation (edge detection) • Minimize with respect to f by Euler equation(image restoration)

17. Zero padding Zero padding Zero padding Convolution Implementation • Neumann boundary conditions • FFT multiplications

18. Deconvolution with Mumford-Shah Regularization blurred suggested edges (v) suggested restoration

19. - The restored image is very sensitive to the recovered kernel. - The recovered kernel depends on the contents of the image. • Suggested: Gaussian kernel parameterized by s. Semi-blind Deconvolution via Mumford-Shah Regularization L. Bar, N. Sochen, N. Kiryati, IEEE Trans. Image Processing, 2006 • Blind deconvolution: the blur kernel is unknown • Chan and Wong 1998:

20. Semi-blind Deconvolution via Mumford-Shah Regularization Chan-Wong suggested method blurred

21. Total Variation Image Deblurring in the Presence pf Salt-and-Pepper noise L. Bar, N. Sochen, N. Kiryati, Scale Space, 2005 (best student paper) • Special care should be taken in the case of salt-and-pepper noise • L2 fidelity term in not adequate anymore

22. Median filter 3x3 window • TV restoration • Noise remains! • Median filter 5x5 window • TV restoration • Nonlinear distortion! Image Deblurring in the Presence pf Salt-and-Pepper noise L. Bar, N. Sochen, N. Kiryati, Scale Space, 2005 (best student paper) • Special care should be taken in the case of salt-and-pepper noise • L2 fidelity term in not adequate anymore • Sequential approach: Deblurring following median-type filtering-poor

23. data fidelity gradients within segments total edge length • Iterate • Minimize with respect to v by Euler equation (edge detection) • Minimize with respect to f by Euler equation(image restoration) Image Deblurring in the Presence pf Salt-and-Pepper noise Suggested approach: robust L1fidelity and Mumford-Shah regularization

24. Linear operator Image Deblurring in the Presence pf Salt-and-Pepper noise Linearization via fixed point scheme: coefficients in nonlinear terms are laggedby one iteration →linear equation

25. blurred blurred and noisy suggested 3x3 median + TV 5x5 median + TV Results - pill-box kernel (9x9), radius 4, 10% noise

26. Results - pill-box kernel (7x7), radius 3, 1% noise blurred and noisy recovered

27. Results - pill-box kernel (7x7), radius 3, 10% noise blurred and noisy recovered

28. Results - pill-box kernel (7x7), radius 3, 30% noise blurred and noisy recovered

29. Theoretical Questions L. Bar, N. Sochen, N. Kiryati, International Journal of Computer Vision • What is the theoretical explanation to the simultaneous deblurring and denoising? • Is Mumford-Shah regularization better than Total Variation? • There is discrimination between image and noise edges. • Image edges are preserved while impulse noise is removed

30. Edge Preservation robust statistics anisotropic diffusion line process (half quadratic) Relations between: • robust statistics • anisotropic diffusion • line process (half-quadratic) were shown by • Black and Rangarajan, IJCV, 1996 • Black, Sapiro, Marimont and Heeger, IEEE T-IP, 1998 Hampel et al., 1986 Perona & Malik, 1987 Geman & Yang, 1993 Charbonnier et al., 1997

31. Edge Preservation Influence function-y Gradient Descent: r’(s)=y(s) r(s) 1. Robust smoothness

32. Edge Preservation • Isotropic diffusion (heat equation) • Anisotropic diffusion (Perona and Malik, 1987) g is “edge stopping” function From robust smoothness point of view Lorentzian 2. Diffusion

33. Diffusion Illustration Original Isotropic Diffusion Anisotropic Diffusion

34. Edge Preservation • Dual function b represents edges • Penalty function Y enforces sparse edges across edges otherwise • From robust smoothness point of view 3. Line-process (Half-Quadratic) (Geman and Yang, 1993)

35. Example: Geman-McClure Function Anisotropic Diffusion Line Process (Half-Quadratic) Robust Smoothing GemanMcClure edge penalty edge stopping function robust r-function

36. Relation to M-S Terms Edges are forced to be smooth and continuous image edges are preserved • The Geman-McClure function in half-quadratic form Appears in M-S terms with b = v2 M-S: extended line process = extended Geman-McClure

37. Color Deblurring in the Presence of Impulsive Noise L. Bar, A. Brook, N. Sochen, N. Kiryati, VLSM’05 • Channels have to be coupled • One edge map for all channels

38. Image Restoration in 3D blurred recovered edges

39. Future Work: Space Variant Image Restoration preliminary results

40. Conclusions Novel unified approach to variational segmentation, deblurring and denoising. Mumford-shah regularization reflects the piecewise-smooth model of natural images. Relations to robust statistics and anisotropic diffusion show that Mumford-Shah regularization is a better edge detector. Restoration outcome is superior to state-of-the-art methods

41. Conclusions Novel unified approach to variational segmentation, deblurring and denoising. Mumford-shah regularization reflects the piecewise-smooth model of natural images. Relations to robust statistics and anisotropic diffusion show that Mumford-Shah regularization is a better edge detector. Restoration outcome is superior to state-of-the-art methods

42. Conclusions Novel unified approach to variational segmentation, deblurring and denoising. Mumford-shah regularizationreflects the piecewise-smooth model of natural images. Relations to robust statistics and anisotropic diffusion show that Mumford-Shah regularization is a better edge detector. Restoration outcome is superior to state-of-the-art methods

43. Conclusions Novel unified approach to variational segmentation, deblurring and denoising. Mumford-shah regularization reflects the piecewise-smooth model of natural images. Relations to robust statistics and anisotropic diffusion show that Mumford-Shah regularization isabetter edge detector. Restoration outcome is superior to state-of-the-art methods

44. Thank you for your attention