Super-Resolution

Super-Resolution Digital PhotographyCSE558, Spring 2003Richard Szeliski

Super-resolution • convolutions, blur, and de-blurring • Bayesian methods • Wiener filtering and Markov Random Fields • sampling, aliasing, and interpolation • multiple (shifted) images • prior-based methods • MRFs • learned models • domain-specific models (faces)- Gary Super-Resolution

Linear systems • Basic properties • homogeneity T[a X] = a T[X] • additivity T[X1+X2] = T[X1]+T[X2] • superposition T[aX1+bX2] = aT[X1]+bT[X2] • Linear system  superposition • Examples: • matrix operations (additions, multiplication) • convolutions Super-Resolution

Signals and linear operators • Continuous I(x) • Discrete I[k] or Ik • Vector form I • Discrete linear operator y = A x • Continuous linear operator: • convolution integral • g(x) = sh(,x) f() d, h(,x): impulse response • g(x) = s h(-x) f() d= [f * h](x)shift invariant Super-Resolution

2-D signals and convolutions • Continuous I(x,y) • Discrete I[k,l] or Ik,l • 2-D convolutions (discrete) • g[k,l] = m,n f[m,n] h[k-m,l-n] • = m,n f[m,n] h1[k-m]h2[l-n] separable • Gaussian kernel is separable and radial • h(x,y) = (22)-1exp-(x2+y2)/2 Super-Resolution

Convolution and blurring Super-Resolution

Separable binomial low-pass filter Super-Resolution

Fourier transforms • Project onto a series of complex sinusoids • F[m,n] = kf[k,l] e-i 2(km+ln) • Properties: • shifting g(x-x0) exp(-i 2fxx0)G(fx) • differentiation dg(x)/dx i 2fxG(fx) • convolution [f * g](x) [F G](fx) Super-Resolution

Blurring examples • Increasing amounts of blur + Fourier transform Super-Resolution

Sharpening • Unsharp mask (darkroom photography): • remove some low-frequency content y’ = y + s (y – g * y)spatial (blur, sharp) freq (blur,sharp) Super-Resolution

Sharpening - result • Unsharp mask: original, blur (σ=1), sharp(s=0, 1, 2) Super-Resolution

Deconvolution • Filter by inverse of blur • easiest to do in the Fourier domain • problem: high-frequency noise amplification Super-Resolution

Bayesian modeling • Use prior model for image and noise • y = g * x + n, x is original, y is blurred • p(x|y) = p(y|x)p(x) = exp(-|y – g*x|2/2σn-2) exp(-|x|2/2σx-2) • -log p(x|y) |y – g*x|2σn-2 + |x|2σx-2where the norm || is summed squares over all pixels Super-Resolution

Parseval’s Theorem • Energy equivalence in spatial ↔ frequency domain • |x|2 = |F(x)|2 • -log p(x|y) |Y(f) – G(f)X(f)|2σn-2 + |X(f)|2σx-2 • least squares solution (∂/∂X = 0)X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] Super-Resolution

Wiener filtering • Optimal linear filter given noise and signal statistics • X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] • low frequencies: X(f) ≈ G-1(f)Y(f)boost by inverse gain (blur) • high frequencies: X(f) ≈ G(f) σn-2σx2 Y(f)attenuate by blur (gain) Super-Resolution

Wiener filtering – white noise prior • Assume all frequencies equally likely • p(x) ~ N(0,σx2) • X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] • solution is too noisy in high frequencies Super-Resolution

Wiener filtering – pink noise prior • Assume frequency falloff (“natural statistics”) • p(X(f)) ~ N(0,|f|-βσx2) • X(f) = G(f)Y(f) / [G2(f) + |f|βσn2/σx2] • greater attenuation at high frequencies G(f) H(f) Super-Resolution

Markov Random Field modeling • Use spatial neighborhood prior for image • -log p(x) = ijCρ(xi-xj)where ρ(v) is a robust norm: • ρ(v) = v2: quadratic norm ↔ pink noise • ρ(v) = |v|: total variation (popular with maths) • ρ(v) = |v|β: natural statistics • ρ(v) = v2,|v|: Huber norm[Schultz, R.R.; Stevenson, IEEE TIP, 1996] i j Super-Resolution

MRF estimation • Set up discrete energy (quadratic or non-) • -log p(x|y) σn-2|y – Gx|2 + ijCρ(xi-xj)where G is sparse convolution matrix • quadratic: solve sparse linear system • non-quadratic: use sparse non-linear least squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …) Super-Resolution

Sampling a signal • sampling: • creating a discrete signal from a continuous signal • downsampling (decimation) • subsampling a discrete signal • upsampling • introducing zeros between samples • aliasing • two sampled signals that differ in their original form (many → one mapping) Super-Resolution

Sampling interpolation Super-Resolution

Nyquist sampling theorem • Signal to be (down-) sampled must have a bandwidth no larger than twice the sample frequency • s = 2 / ns > 2 0 Super-Resolution

Box filter (top hat) Super-Resolution

Ideal low-pass filter Super-Resolution

Simplified camera optics • Blur = pill-box*Bessel2 (diffr.) ≈ Gaussian • Integrate = box filter • Sample = produce single digital sample • Noise = additive white noise Super-Resolution

Aliasing • Aliasing (“jaggies” and “crawl”) is present ifblur amount < sampling (σ = 1) • shift each image in previous pipeline by 1 Super-Resolution

Aliasing - less • Less aliasing (“jaggies” and “crawl”) is present ifblur amount ~ sampling (σ = 2) • shift each image in previous pipeline by 1 Super-Resolution

Multi-image super-resolution • Exploit aliasing to recover frequencies above Nyquist cutoff • kσn-2|yk – Gkx|2 + ijCρ(xi-xj)where Gk are sparse convolution matrices • quadratic: solve sparse linear system • non-quadratic: use sparse non-linear least squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …) • projection onto convex sets (POCS) Super-Resolution

Multi-image super-resolution • Need: • accurate (sub-pixel) motion estimates(Wednesday’s lecture) • accurate models of blur (pre-filtering) • accurate photometry • no (or known) non-linear pre-processing(Bayer mosaics) • sufficient images and low-noise relative to amount of aliasing Super-Resolution

Prior-based Super-Resolution • “Classical” non-Gaussian priors: • robust or natural statistics • maximum entropy (least blurry) • constant colors (black & white images) Super-Resolution

Example-based Super-Resolution • William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,IEEE Computer Graphics and Applications, March/April, 2002 • learn the association between low-resolution patches and high-resolution patches • use Markov Network Model (another name for Markov Random Field) to encourage adjacent patch coherence Super-Resolution

Example-based Super-Resolution • William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,IEEE Computer Graphics and Applications, March/April, 2002 Super-Resolution

References – “classic” • Irani, M. and Peleg. Improving Resolution by Image Registration. Graphical Models and Image Processing, 53(3), May 1991, 231-239. • Schultz, R.R.; Stevenson, R.L. Extraction of high-resolution frames from video sequences. IEEE Trans. Image Proc., 5(6), Jun 1996, 996-1011. • Elad, M.; Feuer, A.. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Proc., 6(12) , Dec 1997, 1646-1658. • Elad, M.; Feuer, A.. Super-resolution reconstruction of image sequences. IEEE PAMI 21(9), Sep 1999, 817-834. • Capel, D.; Zisserman, A.. Super-resolution enhancement of text image sequences. CVPR 2000, I-600-605 vol. 1. • Chaudhuri, S. (editor). Super-Resolution Imaging. Kluwer Academic Publishers. 2001. Super-Resolution

References – strong priors • Freeman, W.T.; Pasztor, E.C.. Learning low-level vision, CVPR 1999, 182-1189 vol.2 • William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, Example-based super-resolution, IEEE Computer Graphics and Applications, March/April, 2002 • Baker, S.; Kanade, T. Hallucinating faces. Automatic Face Gesture Recognition, 2000, 83-88. • Ce Liu; Heung-Yeung Shum; Chang-Shui Zhang. A two-step approach to hallucinating faces: global parametric model and local nonparametric model. CVPR 2001. I-192-8. Super-Resolution

Super-Resolution