Create Presentation
Download Presentation

Download Presentation

Super-Resolution Through Neighbor Embedding

Download Presentation
## Super-Resolution Through Neighbor Embedding

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Super-Resolution Through Neighbor Embedding**Hong Chang, Dit-Yan Yeung and Yimin Xiong Presented By: Ashish Parulekar, Ritendra Datta, Shiva Kasiviswanathan and Siddharth Pal**Contents**• Introduction • What is Super resolution ? • Multiframe superresolution. • Single frame superresolution. • Problem Statement. • Review on Manifold Learning • Experimental Setup • Results • Comments**same resolution**higher resolution Definition of Resolution • Resolution ability to resolve/distinguish two objects • w.r.t image capture devices, resolution is determined by number of sensing elements in the two dimensions • Higher resolution Larger image size but, larger image size higher resolution**superresolution algorithm**high resolution image LR frames Superresolution • Any algorithm/method which is capable of producing an image with a resolution greater than that of the input • Typically, input is a sequence of low resolution (LR) images (also referred to as frames) • LR frames displaced from each other • Have common region of interest**Super Resolution Algorithms**• Multi-frame superresolution • Single-frame superresolution**Multi-frame superresolution**• Introduced by Tsai and Huang • Most common approach to superresolution • Exploits information in several LR frames/images to generate a HR image • Kim, Bose and Valenzuela • Recursive wavenumber domain approach • Basic model employed**Multi-frame superresolution**• Ur and Gross • Ensemble of spatially shifted observations • Papoulis and Brown generalized sampling theorem • Irani and Peleg – Iterative Back Projection • Initial guess of HR image • Compute LR image estimates using imaging process • Back-project the error to improve HR image • S. Lertrattanapanich • Delaunay triangulation of registered images • Surface approximation**Multi-frame superresolution**• Nguyen and Milanfar • Wavelet superresolution • Approximate given points by a sufficiently dense dyadic set • Initially, ignore detail coefficients and obtain regularized least squares solution for approximation coefficients • Use difference between data and approximation to solve for detail coefficients • Reconstruct on HR grid using coefficients**SGWSR 1-D Method**• M. Chappalli and N. K. Bose (2004) • HR reconstruction restricted to LR frames on semi-regular grids • Semi-regular grid tensor product of 1-D arbitrarily irregular grids • LR frames displaced only by translations • approximates far-field imaging, ex: aerial photography, satellite imaging • Based on 1-D lifting – splitting, prediction and updating • Applied on rows and subsequently on columns • Simultaneous noise reduction using hard and soft thresholding of wavelet coefficients**+**odd sj - + sj+1 Pj Uj Uj Pj m u l t i p l e x e r + even dj - + forward transform inverse transform Lifting to construct SGW sj+1**Result of SGWSR 1-D Method**• Results DT bilinear interpolation PSNR: 26.4996 DT bicubic interpolation PSNR: 25.6810 sample LR frame SGWSR soft thresholding PSNR: 28.5462 original**Single-frame superresolution**• Spatial boundedness & Bandlimitedness – not simultaneously possible • exploited to reconstruct higher frequencies • LSI interpolation cannot generate new information • nonlinear and shift-varying interpolation is used • Examples • Directional filters, Adaptive interpolation • Edge-preserving interpolation, perceptual edge enhancement • Learning-based methods • Lack of information – major limitation**Problem Formulation**Training Xsi Training Ysi ? Testing Xt Testing Yt**Manifold Definition Revisited**A manifold is a topological space which is locally Euclidean. Represents a very useful and challenging unsupervised learning problem. In general, any object which is nearly "flat" on small scales is a manifold.**Manifold Learning**• Discover low dimensional structures (smooth manifold) for data in high dimension. • Linear Approaches • Principal component analysis. • Multi dimensional scaling. • Non Linear Approaches • Local Linear Embedding • ISOMAP • Laplacian Eigenmap.**Nonlinear Approaches- ISOMAP**• Construct neighbourhood graph G. • For each pair of points in G, compute, shortest path distances - geodesic distances. • Construct k-dimensional coordinate vectors Geodesic: Shortest curve along the manifold connecting two points**ISOMAP algorithm Pros/Cons**Advantages: • Nonlinear • Globally optimal • Guarantee asymptotically to recover the true dimensionality Drawback: • May not be stable, dependent on topology of data • As N increases, pair wise distances provide better approximations to geodesics, but cost more computation**Local Linear Embedding (a.k.a LLE)**• LLE is based on simple geometric intuitions. • Suppose the data consist of N real-valued vectors Xi, each of dimensionality D. • Each data point and its neighbors expected to lie on or close to a locally linear patch of the manifold.**Steps of locally linearembedding:**• Reconstruction errors are measured by the cost function • έ(W) =**Steps in LLE algorithm**• Assign neighbors to each data point • Compute the weights Wij that best linearly reconstruct the data point from its neighbors, solving the constrained least-squares problem. • Compute the low-dimensional embedding vectors best reconstructed by Wij.**The final step**• Each high-dimensional observation is mapped to a low-dimensional vector representing global internal co ordinates.**Fit locally, Think Globally**From Nonlinear Dimensionality Reduction by Locally Linear Embedding Sam T. Roweis and Lawrence K. Saul**Neighbor embedding method**1) For each patch xqtin image Xt: a) Find the set Nqof K nearest neighbors in Xs. b) Compute the reconstruction weights of the neighbors that minimize the error of reconstructing xqt.**Neighbor embedding methodContd….**c) Compute the high-resolution embedding yqt using the appropriate high-resolution features of the K nearest neighbors and the reconstruction weights. 2) Construct the target high-resolution image Yt by enforcing local compatibility and smoothness constraints between adjacent patches obtained in step 1(c).**Equations to fit**Minimize local reconstruction error Gram Matrix**Solution to Constrained Least Squares Problem**Efficient way 1)Rather than inverting G matrix is to solve the equations Gqwq=1 2)Normailze the weights**Final step**Final equation in the reconstruction of high resolution image And finally, Step 2 of the algorithm is achieved by averaging the feature values in overlapped regions between adjacent patches**Problem Formulation**Training Xsi Training Ysi ? Testing Xt Testing Yt**Intuition**• Patches of the image lie on a manifold Training Xsi Low dimensional Manifold High dimensional Manifold Training Ysi**Algorithm**• Get feature vectors for each low resolution training patch. • For each test patch feature vector find K nearest neighboring feature vectors of training patches. • Find optimum weights to express each test patch vector as a weighted sum of its K nearest neighbor vectors. • Use these weights for reconstruction of that test patch in high resolution.**Experimental Setup**• Target magnification : 3X • Transform to YIQ space • Feature Selection • First derivative of luminance • Second derivative of luminance • Reconstruction of Y and transfer of I and Q from lower dimension.**Results**Experiments with images from the paper. Intuition: Authors’ test set may be biased !**Results**Training Xsi Training Ysi Testing Xt Testing Yt**Results**Training Xsi Training Ysi Testing Xt Testing Yt**Results**Training Xsi Training Ysi Testing Xt Testing Yt**Results**Training Ysi Training Xsi Testing Xt Testing Yt**Results**K=1 K=2 K=3 K=4 k=5 k=6**Results**As expected, good super-resolution images generated. Now for the real test – our images !**Results – their method our data**Our Test Image 1/3X Obtained from scaling down Ground Truth Image**Results – their method our data**High-resolution training images : (A) Similar (B) Dissimilar images Corresponding low-resolution training image**Results – their method our data**We test the results using RMS Error using the following formula: Error = √ (1/n) ∑ || Pground-truth – Pgenerated || This essentially indicates the average Luminance deviations between corresponding pixels in the Ground-truth and Generated Images. Remember, the Chrominance components I and Q were copied into the generated image without change.**Results – Similar training Image**K=1 K=2 K=3 K=4 K=5 K=6**Results – Different training Image**K=1 K=2 K=3 K=4 K=5 K=6**Comments on results**• Best results when using • Different images: K = 4 or 5 (confirming to what is stated in the paper). • Same images: K =1 (why ?))**Comments**• Method worked well with our test data. • Care should be taken about image patches which do NOT form a manifold. • Limitations on magnification. • Size/ Overlap in the patch. • Many extensions are possible: • Diagonal gradients for LR (increased dimensionality) • Using Lumina as feature vector in LR • Can be extended to Video shot SR • Multiple training images