1. Image Stitching Shangliang Jiang Kate Harrison

2. What is image stitching?

3. What is image stitching?

4. Introduction • Are you getting the whole picture? • Compact Camera FOV = 50 x 35°

5. Introduction • Are you getting the whole picture? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135°

6. Introduction • Are you getting the whole picture? • Compact Camera FOV = 50 x 35° • Human FOV = 200 x 135° • Panoramic Mosaic = 360 x 180°

7. Recognizing Panoramas • 1D Rotations (q) • Ordering  matching images

8. Recognizing Panoramas • 1D Rotations (q) • Ordering  matching images

9. Recognizing Panoramas • 1D Rotations (q) • Ordering  matching images

10. 2D Rotations (q, f) • Ordering  matching images Recognizing Panoramas • 1D Rotations (q) • Ordering  matching images

11. 2D Rotations (q, f) • Ordering  matching images Recognizing Panoramas • 1D Rotations (q) • Ordering  matching images

12. 2D Rotations (q, f) • Ordering  matching images Recognizing Panoramas • 1D Rotations (q) • Ordering  matching images

13. Recognizing Panoramas

14. Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

15. Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

16. Overview • Feature Matching • SIFT Features • Nearest Neighbor Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

17. SIFT Features • SIFT features are… • Geometrically invariant to similarity transforms, • some robustness to affine change • Photometrically invariant to affine changes in intensity

18. Overview • Feature Matching • SIFT Features • Nearest Neighbor Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

19. Nearest Neighbor Matching • Find k nearest neighbors for each feature • k  number of overlapping images (we use k = 4) • Use k-d tree • k-d tree recursively bi-partitions data at mean in the dimension of maximum variance • Approximate nearest neighbors found in O(nlogn)

20. Overview • Feature Matching • SIFT Features • Nearest Neighbor Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

21. Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

22. Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

23. Overview • Feature Matching • Image Matching • Random Sample Consensus (RANSAC) for Homography • Probabilistic model for verification • Bundle Adjustment • Image Compositing • Conclusions

24. Overview • Feature Matching • Image Matching • Random Sample Consensus (RANSAC) for Homography • Probabilistic model for verification • Bundle Adjustment • Image Compositing • Conclusions

25. RANSAC for Homography

26. RANSAC for Homography • SIFT features of two similar images

27. RANSAC for Homography • SIFT features common to both images

28. RANSAC for Homography • Select random subset of features (e.g. 6) • Compute motion estimate • Apply motion estimate to all SIFT features • Compute error: feature pairs not described by motion estimate • Repeat many times (e.g. 500) • Keep estimate with best error

29. RANSAC for Homography

30. Overview • Feature Matching • Image Matching • Random Sample Consensus (RANSAC) for Homography • Probabilistic model for verification • Bundle Adjustment • Image Compositing • Conclusions

31. Probabilistic model for verification • Potential problem: • Two images don’t match… • … but RANSAC found a motion estimate • Do a quick check to make sure the images do match • MAP for inliers vs. outliers

32. Finding the panoramas

33. Finding the panoramas

34. Finding the panoramas

35. Overview • Feature Matching • Image Matching • RANSAC for Homography • Probabilistic model for verification • Bundle Adjustment • Image Compositing • Conclusions

36. Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

37. Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

38. Overview • Feature Matching • Image Matching • Bundle Adjustment • Error function • Image Compositing • Conclusions

39. Overview • Feature Matching • Image Matching • Bundle Adjustment • Error function • Image Compositing • Conclusions

40. Bundle Adjustment • New images initialised with rotation, focal length of best matching image

41. Bundle Adjustment • New images initialised with rotation, focal length of best matching image

42. Error function • Sum of squared projection errors • n = #images • I(i) = set of image matches to image i • F(i, j) = set of feature matches between images i,j • rijk = residual of kth feature match between images i,j • Robust error function

43. Overview • Feature Matching • Image Matching • Bundle Adjustment • Error function • Image Compositing • Conclusions

44. Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

46. Overview • Feature Matching • Image Matching • Bundle Adjustment • Image Compositing • Conclusions

47. Blending

48. Gain compensation

49. How do we blend? Linear blending Multi-band blending

50. Multi-band Blending • Burt & Adelson 1983 • Blend frequency bands over range l