Sky Finder: Attribute-based Sky Image Search

1. Sky Finder: Attribute-based Sky Image Search Litian Tao Lu Yuan Jian Sun SIGGRAPH 2009 Hong Kong University of Science and Technology Beihang University Microsoft Research Asia

4. Difficult to capture a beautiful sky • Dynamic rang of scene > dynamic rang of camera • Good timing is very important

5. Need to search a sky image • Sky replacement • Background images for composition • 2D design, film production, image editing

6. Frustrating using search engines Bing Image Search Google Image Search

7. Our System • Search desired sky images from a sky dataset 500,000 sky images from Flickr

8. Related Work – Image as Query • Scene Completion [Hays and Efros 2007] • Face Swapping [Bitouk et al. 2008] • Photo Clip Art [Lalonde et al. 2008]

9. Related Work – Text as Query • Semantic Photo Synthesis [Johnson et al. 2006]

10. Challenges • Image-query based approahces • Finding a good image example is also a search problem. • Difficult for interactive search in a large dataset • Text -query based (Semantic Photo Syn/Image Search E) • Only can find images including sky • Without further control

11. Our solution - Attribute based search • Semantic attributes • Category • Layout • Horizon height • Sun existence/position • Richness • A large sky database

12. Data Collection • Images from Flickr. com • High quality photos • Human-labeled tags sky cloudy sunset sunrise storm colourskies southfloridasky beautyofsky … sky&clouds 5 keywords 95 user groups 1.3 million images

13. Attributes – Category • 3 Categories • blue-sky • cloudy-sky • sunset • 2000 training images blue-sky cloudy-sky sunset uncertain

14. Bag-of-Word Representation Feature: SIFT + Color Step1: Codebook Generation Feature Space Partition Feature Quantization Randomized Forests [Moosmann et al. 2006] 2:Representation bag of codewords

15. Attributes – Category cloudy-sky SVM blue-sky SVM sunset SVM bag of blue-sky words blue-sky score bag of cloudy-sky words cloudy score bag of sunset words sunset score

16. Attributes • Category • Layout • Horizon height • Sun existence/position • Richness Defined on sky region segmentation

17. Sky Region Segmentation Test Training Three Sky/non-sky pixel classifiers blue-sky cloudy-sky Graph-cut segmentation sunset

18. Attributes – Layout & Horizon full-sky object-in-sky landscape normal-sky others A>95% 95%>A>70% A<70% : Bounding Box –cover 95% sky pixels # sky pixels A : in the bounding box # pixels : Horizon height

19. Attributes – Sun luminance channel magenta channel in CMYK sunset image Bright Region Extraction Shape filter sun mask

20. Attributes – Richness Images Sky region Edge map Richness

21. Attribute Accuracy • Test dataset • 6, 000 images • Performance Precision = Recall = # trueclassifiedblue-sky images # true classifiedblue-sky images # classifiedblue-sky images # total blue-sky images

22. Experimental Evaluation • Test dataset • 6, 000 images • Performance Precision = in blue-sky Recall = in blue-sky # true detectedsky pixels # truedetectedsky pixels Both sky and non-sky region are gloomy # detectedsky pixels # total sky pixels

23. Experimental Evaluation • Test dataset • 6, 000 images • Performance Precision = in sunset Recall = in sunset # true detectedsuns # truedetectedsuns # detectedsuns # total suns Sun is largely occluded by clouds

24. Color based re-ranking • Sky color representation • Color signature: • Similarity • Earth Mover’s Distance (EMD) (a) sunset + landscape + horizon + sun position after color-based re-ranking (b)

25. User Interface Category SVM scores (3D)

26. User Interface Horizon and sun canvas Category Triangle blue-sky Layout Richness landscape cloudy-sky sunset normal-sky full-sky after PCA object-in-sky others

27. Demo

28. Path Search • How to find such a sky image? Our solution: sky graph + smoothed path ?

29. Graph Construction • Building a graph is difficult • Pairwise distance computation is expensive • Semantic metric is required • Sparse graph using attributes • 1: use categoryand richness attributes -> 2000 candidates • 2: re-rank candidates by color ->Top 200 neighbours • 3: use color similarity (EMD) as edge weight

30. Path Search max-transition-cost = 2 • Finding a path • Shortest path • Our path 2 B C 2 2 A D 5 max-transition-cost = 5

31. Sky Replacement • Need to change the foreground color • Sometimes not visually plausible • Our method: category-specific color transfer Apply learned transfer from(s0,s1) to foreground original image retrieved image blue-sky a sky_o sky b ? non-sky_o non-sky cloudy-sky

32. Sky Replacement • Direct cut-and-paste? • Sometimes not visually plausible • Need to change the foreground color • How to compute the color transfer variables? Apply b on foreground original image retrieved image a sunset sky sky b ? non-sky non-sky

33. Demo

34. Summary • Semantic level search • Semantic attributes • Intuitive user interface • Path finding • Very efficient and scalable • Image search -> attributes -> text search

35. On-line Demo • http://jiansun5/SkyFinderEntry/

36. Questions?

37. Sun in blue-sky category • Ratio of images containing sun in three categories • blue-sky: 0.9% • cloudy-sky: 10.0% • sunset: 25.6%.

38. Other attributes • Common attributes in our half a million dataset • Category, layout, horizon height, sun, richness • Other potential attributes: • Lightening • Moon • Polar light • … • We will add these attributes as the database size grows.