Image Resolution Improvement from Multiple Images

1. Image Resolution Improvement from Multiple Images Donald Bailey Institute of Information Sciences and Technology Massey University Palmerston North NEW ZEALAND

2. Overview • Describe the resolution improvement process • Describe the results of my investigations • Investigations in 1 dimensions • Super-resolution of bar codes • Registration in 2 dimensions • Comparison of registration methods • Detailed description of predictive interpolation

3. Description of the Problem • Given a set of related independent low resolution images, combine these together to construct a single high resolution image • output has more detail than any of the input images

4. Resolution Improvement • Resolution limited by number of pixels • Resolution depends on sampling density • An ensemble of images: • Each image provides separate samples • Potentially higher sampling density • Reconstruction steps: • Register images • Resample ensemble • Inverse filter

5. Sampling Requirements • Images must be sub-sampled • If sample rate is greater than Nyquist rate • Can reconstruct the image at any desired resolution • A single image contains all information • Can only improve the signal to noise ratio • If sample rate is less than Nyquist rate • Each individual image is aliased • Cannot obtain higher real resolution from single image • Resampling the ensemble untangles the aliased information

6. One Dimension Example “Super-Resolution of Bar Codes”, D.G. Bailey, Journal of Electronic Imaging, 10 (1), pp 213-221 (January 2001). • Problem: How can we read this bar code?

7. 6 digits 6 digits guard bands Information content of UPC Bar Codes • 12 digits, each 7 units wide • Guard bands • Total width is 95 units • Each bar or space is 1-4 units wide • Broadband frequency spectrum • Centre of main lobe contains required data

8. Super-resolution procedure • A tilted 2D bar code image provides an ensemble of independently sampled 1D images • Register low-resolution images • gives relationship between individual images • Resample the ensemble at a higher rate • creates a high resolution image • Remove system effects • reduces the sampling blur and effect of camera electronics

9. Input image Phase image Registration • Determines the offset between rows • Phase shift in frequency domain proportional to linear offset and spatial frequency • Procedure: • Fourier Transform each row, keep phase • Subtract phase of first row from each row • Unwrap phase image • Discard higher frequencies • Least squares fit to calculate offset per row

10. Resampling • By interleaving samples from different rows increase the sample rate. Original image samples Selectedsamplerows New image samples

11. Resampling • By interleaving samples from different rows increase the sample rate. Original image samples Selectedsamplerows New image samples

12. Resampling • Increase sample rate by an integer multiple of original sample rate • Select rows with offsets nearest the desired sample positions 4 x sample rate

13. Amplitude - w - w w w / 2 / 2 0 0 0 0 0 Spatial frequency Amplitude - w - w w w / 2 / 2 0 0 0 0 0 Spatial frequency Practical limitations From synthetic image From actual image

14. Practical limitations • Real bar code limitations • Ink smearing means bar and space widths not exact • Smears the envelope in the frequency domain • Image capture degradations Lens system Camera angle Sensor Digital Image Video signal Cameraelectronics Videoframegrabber Object

15. Practical limitations • Image distortions • Perspective distortion from camera angle • Lens distortion • Lens point spread function • Spatially variant low pass filter • Image sensor • area sampling - low pass filter • aliasing • Camera electronics • low pass filter, perhaps with high frequency emphasis • Frame grabber • sampling (more aliasing)

16. Removing system effects • Aliasing is not a problem • it is actually necessary for higher resolution reconstruction • resampling the ensemble untangles the aliased information • Main effects are the low pass filter characteristics • lens point spread function • area sampling in sensor • smoothing filter camera electronics • anti-alias filter in frame grabber

17. Amplitude - w - w w w / 2 / 2 0 0 0 0 0 Spatial frequency Removing system effects • Assume no distortion, and no spatial variation in the low pass filter characteristic • Estimate the system response by comparing synthetic and actual reconstructed images • Remove using an inverse linear filter

18. Results Original image Resampled ensemble System response removed Straightened and averaged Thresholded

19. Bar Code Conclusions • A two-dimensional image of a bar code tilted slightly provides an ensemble of related one-dimensional images • The low resolution images must be aliased • It is necessary to compensate for limitations in the image capture system • Analogue video cameras make more complex • Image sampled twice • Additional analogue filters • Modest gains in resolution are achievable

20. Extending to 2 Dimensions • Problem is considerably more complex • Require multiple 2 dimensional images • Captured at different times • Motion is a limitation • Registration more complex • Bar code images all had constant offset per row • In 2D every image is independent, with 2D offset • Resampling more complex • Need more images for same improvement • 4 images to improve resolution by 2

21. Registration in 2 Dimensions • Requirements • Accurate sub-pixel offset between images • Work directly on low resolution images • Insensitive to aliasing • Tolerates a low level of noise • Does not rely on particular objects in the image • For practical use, must be fast • Conventional approach to sub-pixel registration • Interpolate images to chosen high resolution • Increased data volume slows this method down • Perform pixel accuracy registration • Requires a search

22. Sub-Pixel Registration Methods • Phase based methods • Similar to 1D case, but extended to 2D • Determine fit surface on integer grid • Interpolate this to find optimum fit • Correlation methods • Difference methods • Predictive interpolation • A new method that turns problem around • Other approaches • Rely on locating objects or edges within the image

23. Phase Methods • An offset in the image domain corresponds to a phase shift in the frequency domain • Procedure • Window the image and reference • FFT and keep the phases • Unwrap the phase difference • Weighted least squares fit of a plane to the phase

24. Correlation - Pixel Accuracy • Multiply an offset image by a reference • Accumulate product in the overlap region • Normalise by the average pixel value • Prevents bias if the image has a gradient • Frequency domain correlation

25. c ( i ) Correlation peak c 0 c 1 c i - 1 pk i i i i - 1 0 1 Correlation - Sub-pixel Accuracy • Perform pixel level correlation first • Interpolate to find peak to sub-pixel accuracy • Expect correlation peak to be a pyramid • Only strictly true with regions of uniform value • Approximately true if there are step edges • Width of pyramid is twice smallest feature width • Only local information should be used

26. Difference Methods • Can be shown to be related to correlation • Subtract an offset image from a reference • Accumulate difference in the overlap region • Minimum gives offset to nearest pixel • For sub-pixel accuracy • Expect minimum to be an inverted pyramid (locally) • Interpolate to find minimum using previous method

27. Other Registration Methods • Centre of gravity • Segment objects from background • Centre of gravity of objects to sub-pixel accuracy • Line fitting • Detect lines or edges • Fit a line or curve to detected points • Requires knowledge of contents of image • Accuracy limited by size of object / edge and accuracy of segmentation / detection

28. Predictive Interpolation • Turns the problem around • Predicts the pixel values as a function of those in a reference image • Uses the bilinear interpolation equation as a linear predictor • Subject to the constraint: • Then offset is:

29. Predictive Interpolation • Procedure • Requires the image to be pre-registered to nearest pixel • Can use a search, or hierarchical registration to do this • Determine coefficients Axx that minimise the error • Uses weighted least squares • Weight each point with standard deviation of its 4 references • From coefficients, get offset directly • Properties • Very fast - single pass if registered to nearest pixel • Very good accuracy - typically better than 5% of a pixel

30. Evaluating the Registration • Requires a set of images with known offsets • Start with a single high resolution image • Simulate capturing with a lower resolution camera by filtering and subsambling • Add random Gaussian noise to each low resolution image

31. Evaluation Procedure • Measure offset between each pair of images • 36 offset measurements for 9 images • Enforce consistency between measurements • Only 8 independent offsets • Will reduce errors up to • Error is difference between expected offset and measured offset • Average to give the RMS registration error

32. A 0.15 Correlation Difference Phase Predictive 0.10 RMS Error 0.05 0.00 0 5 10 15 20 25 30 35 40 Noise SD Results • Structured image • No noise: • All worked well • Noise sensitivity: • Phase methodonly few frequencies • Predictive methodweighting gives onlyfew measurement

33. B 0.15 Correlation Difference Phase Predictive 0.10 RMS Error 0.05 0.00 0 5 10 15 20 25 30 35 40 Noise SD Results • Low detail image • No noise • Phase is best • Others adequate • Noise sensitivity: • Predictive methodsensitive to noise

34. C 0.15 Correlation Difference Phase Predictive 0.10 RMS Error 0.05 0.00 0 5 10 15 20 25 30 35 40 Noise SD Results • Medium detail image • No noise • Phase and Predictivesignificantly better • Noise sensitivity: • Correlationnoise insensitive • Differencenoise insensitive

35. 0.15 Correlation Difference Phase Predictive D 0.10 RMS Error 0.05 0.00 0 5 10 15 20 25 30 35 40 Noise SD Results • Low detail text • No noise • Phase and predictivesignificantly better • Noise sensitivity: • Predictive methodimproves with lownoise

36. 0.50 Correlation Difference 0.40 Phase Predictive E 0.30 RMS Error 0.20 0.10 0.00 0 5 10 15 20 25 30 35 40 Noise SD Results • High detail text • Significant aliasing • No noise • Correlation andDifference aresignificantly poorer • Phase and Predictivegive excellentresults

37. Summary of Results • Correlation and Difference methods • Similar results • Performance deteriorates with increasing detail • Relatively insensitive to noise • Phase and Predictive methods • Best overall with 1 - 2 % pixel accuracy • Includes enforcing consistency • Predictive method • Results almost independent of image • Improved with addition of small amounts of noise

38. Reconstruction Resampled Low resolution Inverse filtered

39. Conclusions of Comparison • Correlation and difference methods • Insensitive to noise • Perform poorly in presence of high detail • Pyramidal interpolation method breaks down • Phase and predictive methods • More sensitive to noise • Suitable for registration for resolution improvement • Predictive method less expensive than phase

40. A A B B C C D D E E Detailed Properties of Predictive Method • Two components to the errors • Systematic component • Random component

41. Effect of Match Window Size • Systematic bias • No significant change above 20 x 20 pixels • Bias is therefore inherent in the method used • Random component • Depends significantly on window size for small windows • When random component has approximately same magnitude as the bias, it stops improving • No further improvement above about 100 x 100 pixels • Above this, the error has distinct double peak • Random component is therefore limited by systematic bias

42. Summary of Predictive Registration • Very fast • Requires only a single pass through the image • (2 passes if not already registered to nearest pixel) • Accuracy is 4 - 5% of pixel • This is between pairs of images • May be improved by enforcing consistency • Optimum match window size is about 100 x 100 pixels • Robust to moderate levels of noise • Relatively insensitive to aliasing • Suitable for images captured in identical conditions • Sensitive to contrast and brightness

43. Overall Summary • Described resolution improvement • Reconstruction steps: registration, resampling, inverse filtering • Examined some of the preconditions: aliasing • Discussed some of the limitations: lens and camera blurring • Presented results from 1D • Super-resolution of bar code images • Presented results from 2D registration • Comparison of registration methods • Description of a new fast method • Predictive Interpolation method

44. References • “Super-resolution of bar-codes”, D.G. Bailey, Journal of Electronic Imaging, vol 10 (1), pp 213-221 (2001) • “Predictive Interpolation for Registration”, D.G. Bailey, Proceedings of Image and Vision Computing Conference NZ, pp 240-245 (November 2000) • “Image Registration Methods for Resolution Improvement”, D.G. Bailey and T.H. Lill, Proceedings of Image and Vision Computing NZ, pp 91-96 (August 1999) • “Super-Resolution of Bar Codes”, D.G. Bailey, SPIE Proceedings, Vol 3521 Machine Vision Systems for Inspection and Metrology VII, Boston, pp 204-213 (November 1998)