Scene Completion Using Millions of Photographs

1. Scene Completion Using Millions of Photographs James Hays, Alexei A. Efros Carnegie Mellon University ACM SIGGRAPH 2007

2. Outline • Introduction • Overview • Semantic Scene Matching • Local Context Matching • Results and Comparison • Conclusion

3. Outline • Introduction • Overview • Semantic Scene Matching • Local Context Matching • Results and Comparison • Conclusion

4. Introduction • Image completion(inpainting, hole-filling) • Filling in or replacing an image region with new image data such that the modification can not be detected

5. Introduction • The data could have been there • The data should have been there

6. Introduction • The existing methods operate by extending adjacent textures and contours into the unknown region • Filling in the unknown region with content from the known parts of the input image

7. Introduction • The assumption is that all the necessary image data to fill in an unknown region is located somewhere else in the same image • This assumption is flawed

8. Outline • Introduction • Overview • Semantic Scene Matching • Local Context Matching • Results and Comparison • Conclusion

9. Overview • We perform image completion by leveraging a massive database of images • Two compelling reasons • A region will be impossible to fill plausibly using only image data from the source image • Reusing that content would often leave obvious duplications

10. Overview • There are several challenges with drawing content from other images • Computational • Semantically invalid • Seamlessly

11. Overview • Alleviate computational and semantic • Find images depicting semantically similar scenes • Use only the best matching scenes to find patches which match the content surrounding the missing region • Seamlessly combine image regions • Graph cut segmentation • Poisson blending

12. Outline • Introduction • Overview • Semantic Scene Matching • Local Context Matching • Results and Comparison • Conclusion

13. Semantic Scene Matching • Our image database • Download images in thirty Flickr.com groups • Download images based on keyword searches • Discarded duplicate images and images that are too small • Distributed among a cluster of 15 machines • Acquir about 2.3 million unique images

14. Semantic Scene Matching • Look for scenes which are most likely to be semantically equivalent to the image requiring completion • GIST descriptor • Augment the scene descriptor with color information of the query image down-sampled to the spatial resolution of the gist

15. Semantic Scene Matching • Given an input image to be hole-filled, we first compute its gist descriptor with the missing regions excluded • We calculate the SSD between the the gist of the query image and every gist in the database • The color difference is computed in the lab color space

16. Outline • Introduction • Overview • Semantic Scene Matching • Local Context Matching • Results and Comparison • Conclusion

17. Local Context Matching • Having constrained our search to semantically similar scenes we can use Template matching to more precisely align

18. Local Context Matching • Pixel-wise alignment score • We define the local context to be all pixels within an 80 pixel radius of the hole’s boundary • This context is compared against the 200 best matching scenes • Using SSD error in lab color space

19. Local Context Matching • Texture similarity score • Measure coarse compatibility of the proposed fill-in region to the source image within the local context • Computed as a 5x5 median filter of image gradient magnitude at each pixel • The descriptors of the two images are compared via SSD

20. Local Context Matching • Composite each matching scene into the incomplete image at its best placement using a form of graph cut seam finding and standard poisson blending

21. Local Context Matching • Past image completion algorithms • The remaining valid pixels in an image can not changed • Our completion algorithms • Allow to remove valid pixels from the query image • But discourage the cutting of too many pixels

22. Local Context Matching • Past seam-finding • Minimum intensity difference between two images • Cause the seam to pass through many high frequency edges • Our seam-finding • Minimum the gradient of the image difference along the seam

23. Local Context Matching • We find the seam by minimizing the following cost function • : unary costs of assigning any pixel p, to a specific label L(p) • L(p) : patch or exist

24. Local Context Matching • For missing regions of the existing image • is a very large number • For regions of the image not covered by the scene match • is a very large number • For all other pixels • is pixel’s distance from the hole • k = 0.02

25. Local Context Matching • is non-zero only for immediately adjacent, 4-way connected pixels • L(p) = L(q), the cost is zero • L(p) L(q), • is the magnitude of the gradient of the SSD between the existing image and the scene match at pixels p and q

26. Local Context Matching • Finally we assign each composite a score • The scene matching distance • The local context matching distance • The local texture similarity distance • The cost of the graph cut • We present the user with the 20 composites with the lowest scores

27. Local Context Matching

28. Outline • Introduction • Overview • Semantic Scene Matching • Local Context Matching • Results and Comparison • Conclusion

29. Results and Comparison

30. Results and Comparison

31. Results and Comparison

32. Results and Comparison

33. Results and Comparison • Lucky • Find another image from the same physical location • It is not our goal to complete scenes and objects with their true selves in the database

34. Results and Comparison

35. Results and Comparison

36. Results and Comparison • Failure cases : artifact

37. Results and Comparison • Failure cases : semantic violations

38. Results and Comparison • Failure cases : no object recognition

39. Results and Comparison • Failure cases : past methods perform well • For uniformly textured backgrounds • Our method is unlikely to find the exact same texture in another photograph

41. Outline • Introduction • Overview • Semantic Scene Matching • Local Context Matching • Results and Comparison • Conclusion

42. Conclusion • This paper • Present a new image completion algorithm powered by a huge database. • Unlike past methods that reuse visual data within the source image. • Further work • Two million images are still a tiny fraction of the high quality photograph available. • Our approach would be an attractive web-base application.

43. Thank you!!!