A Sampled Texture Prior for Image Super-Resolution

1. Text truth Text low-res Text training image Brick truth Brick low-res Central pixel Brick training image Beads truth Beads low-res Beads training image Number of images Sample texture Original image Noise (grey levels) = 0.01 = 0.04 Number of images 5 5 10 10 20 20 0.5 0.5 2.5 2.5 7.5 7.5 10 10 Noise (grey levels) = 0.64 = 0.16 A Sampled Texture Prior for Image Super-Resolution Lyndsey C. Pickup, Stephen J. Roberts and Andrew Zisserman, Robotics Research Group, University of Oxford A few results Introduction The Sample-Based Prior Goal: Super-resolution aims to produce a high-resolution image from a set of low-resolution images by recovering or inventing plausible high-frequency information. With more than one low-resolution image, high-frequency information can be recovered by making use of sub-pixel image displacements. Given some basic knowledge about the image subject, it should be possible to improve the super-resolution estimate by incorporating information available from similar images. For each image type, nine datasets were generated, using varying numbers of inputs and different noise levels. For each dataset, the best of several runs (while varying free model parameters) is shown. Note that training images do not overlap with image sections used as ground truth. An approach that has been shown to work well in texture synthesis is used to build the image prior. Rather than developing a parametric model, samples are taken from other images. Small sample image patches of edge length 5 to15 pixels are taken from a training image containing similar textures to the images being super-resolved. Low-resolution images (inputs) The goal is to reconstruct the high-resolution image to the right. It is assumed that the low-resolution images have been generated from this image through a process that blurs and subsamples the high-resolution image, and adds Gaussian noise. For each patch, neighbourhood information is stored along with a central pixel value. The patches are weighted by a 2D Gaussian so pixels near the centre carry more weight. Pixel neighbourhood The model • To evaluate this texture-based prior • Take each pixel xi in the super-resolution image x, and find its neighbourhood R(xi). • Find the closest matching neighbourhood in the patches sampled from the training image. • Look up the central pixel value of that neighbourhood, LR(xi). • Assume xi is Gaussian distributed about LR(xi) with precision T. • This gives us: Image x. Warp, with parameters . Blur by point-spread function. Decimate by zoom factor. Corrupt with additive Gaussian noise. There are K low-resolution images y(k), generated by y(k) = W(k)x + G(k), where G is a vector of i.i.d. Gaussians such that G»N(0,-1), and G is the noise precision. If x has N pixels and each y(k) has M pixels, W(k) is an M£N matrix which encompasses the warping, blurring and decimation of x to produce y(k). We denote the registration and blurring parameters of y(k) by (k). Maximum Likelihood Solution A plot of root mean square (RMS) errors obtained from texture-based prior against those from the Huber-MAP images, where the error is calculated with respect to the ground truth images. In all cases, the texture-based prior’s image has a lower RMS error. The maximum likelihood images are very noisy because the problem can be ill-conditioned. • Solving for the super-resolution image • Start with an initial estimate of x, for instance the average image. We assume that the image registration has already been estimated accurately. • Sample patches from an image in the same class as the super-resolution image being recovered. Store the neighbourhood and patch centre data for use with the texture-based prior. • Use SCG to optimize the MAP version of the super-resolution model incorporating the new p(x): Maximum A Posteriori Solution where g is an approximation to the image gradient at each pixel in four different directions (horizontal, vertical, both diagonals), and  is the Huber function, defined Using the wrong texture With a weak prior, the input data contains enough evidence for the prior to look like the ground truth, but as  increases, the prior term dominates. Since this is an image of a fairly specific type, it should be possible to do better. Rather than using a generic Huber-style prior based on image statistics, samples from similar images can be used to build a more appropriate prior. where  encompasses the T and G terms from the prior and data likelihood.