Presentation Transcript

1. Web Image Prediction Using Multivariate Point Processes

   Gunhee Kim1Li Fei-Fei2 Eric P. Xing1 1: School of Computer Science, Carnegie Mellon University 2: Computer Science Department, Stanford University August14, 2012
2. Outline Problem Statement Method Multivariate Point Process + Poisson Regression Full model of Intensity Function Learning and Prediction Personalization Experiments Conclusion
3. Outline Problem Statement Method Multivariate Point Process + Poisson Regression Full model of Intensity Function Learning and Prediction Personalization Experiments Conclusion
4. Problem Statement - Web Image Prediction A photo stream of world+cupfrom Flickr up to 12/31/2008. Each image is associated with meta-data (timestamp, owner ID). Can we guess what photos will appear on the Flickr at tq = 6/6/2009? Actual images at tq Collective Image prediction Actual images by uqat tq PersonalizedImage prediction
5. Why is Image Prediction Interesting? Predicting User Behaviors on the Web User behavior on the Web changes over time. (1) Keyword search What query terms are popular? What documents are most relevant? What documents are likely to be clicked? (2) News recommendation (3) Product search Few previous work on what images people are interested in. [D08] Dakka et al. CIKM 2008 [M09] Metzler et al. SIGIR 2009 [K10] Kulkani et al, WSDM 2011 [V11] Amodeo et al, CIKM2011 [R12] Radinsky et al, WWW 2012
6. Why is Image Prediction Interesting? Time-sensitive Image Reranking Submit the term world+cupinto Google/Bing/Flickr engines Google Bing Severely redundant. Almost identical all year long. Any meaningful order? Increase diversity by temporal trends Flickr Ranking by temporal suitability
7. Why is Image Prediction Interesting? Time-sensitive Image Reranking Time-sensitive image reranking For tq = Jun. 23 (summer) Personalized Time-sensitive image reranking For tq = Aug. 23 and uq= 15655191 For tq = Feb. 5 (winter)
8. Relation to Previous Work Web Content Dynamics Similar Image Retrieval Semantic meaning of keyword + feature-wise similarity Text based method [A11,W06] Image-based method [K10] [D11, P08, T08] No image prediction No personalization Temporal trends + user histories Leveraging Web Photos to Infer Missing Information Image basedCollaborative Filtering Scene completion [H07] 3D models of landmarks [SN10] Semantic image hierarchy [L10] Social trends in politics and market [J10] Spatio-temporal events [S10] Future images: not studied as missing info to be inferred. Images: source of predictionnot subject of prediction [A11] Ahmed et al. AISTAT11 [W06] Wang et al. KDD06 [K10] Kim et al, ECCV10 [D11] Deng et al. CVPR 11 [P08] Dhilbin et al. CVPR08 [T08] Torralba et al. PAMI08 [SN10] Snavely et al. IEEE10 [L10] Li et al. CVPR10 [J10] Jin et al. MM10 [S10] Singh et al. MM10 [H07] Hayes et al. SIGGRAPH07
9. Summary of Contribution Collective and Personalized Web Image Prediction Few previous work for large-scale Web images. (1) Predicting user behaviors on the Web (2) Time-sensitive image reranking (2) News recommendation Algorithm based on multivariate point process Novel in image retrieval literature Flexibility, optimality, scalability, and prediction accuracies More than 10 million images of 40 topics Outperform baselines (PageRank based IR, Topic modeling)
10. Outline Problem Statement Method Multivariate Point Process + Poisson Regression Full model of Intensity Function Learning and Prediction Personalization Experiments Conclusion
11. Multivariate Point Process (MPP) A stochastic process that consists of a series of random events in time and spaces. Neural spiking modeling Geology Ecology Micro-earthquake data [Schoenberg] Locations of Lauraceaetrees [Moller et al. 2008] [Brown et al. Nat.Neuro.04] Computer Vision Statistical Model for spatio-temporal events Events in video [Prabhakar et al. CVPR10] Crowd counting [Ge et al.CVPR08]
12. MPP for Image Streams An occurrence of a particular image at a particular time = A point in time and image space A short stream of penguin images v1 : ice hockey v2 : animal penguin v3 : snowy mountain Each image is associated with (visual cluster, timestamp) Discrete-time trivariate PP
13. Mathematical Formulation for MPP A short stream of penguin images Intensity function for VC i att Infinitesimal expected occurrence rate of visual cluster iat time t Covariates: any likely factors to be associated with image occurrences (ex. Time, season, and other external events) The intensity function is represented by exponential of linear covariate functions. : Parameter set : covariate function
14. MLE solution for MPP A short stream of penguin images Poisson regression Parametric form of intensity functions with covariates Log-likelihood of an observed stream Globally-optimal solution MLE solution can tell which covariates are contributing for the occurrence of visual cluster i
15. Sparse MLE solution for MPP A short stream of penguin images Log-likelihood of an observed stream L1 (Lasso) penalty A sparse solution is encouraged For each visual cluster, only a small number of strong factors affect image occurrence. MLE solution: Cyclic coordinate descent [Friedman et al. 2010].
16. A Toy Example of Image Prediction Covariates: only year and months (1 + 7 + 12 = 20 parameters) Shark example Peaked in summer (Sea tour) Peaked in January (Ice hockey) Every month Observed occurrence data Every year
17. Outline Problem Statement Method Multivariate Point Process + Poisson Regression Full model of Intensity Function Learning and Prediction Personalization Experiments Conclusion
18. Full Model of Intensity Functions History component Correlation component Externalcomponent Any probable factors can be included without performance loss because we encourage a sparse solution.
19. Full Model of Intensity Functions History component Correlation component Externalcomponent Linear autoregressive (AR) process of order P Typical pattern ofannual periodicity Biphasic = bursty occurrence
20. Full Model of Intensity Functions History component Correlation component Externalcomponent Existence or absence of a VC can be a strong clue. Synchronized 4 months lag
21. Full Model of Intensity Functions History component Correlation component Externalcomponent Month covariate User covariate Note 1. Flexibly add or remove covariate functions according to the characteristics of image topics. 2. AR can be replaced by a more general temporal model such as ARMA.
22. Outline Problem Statement Method Multivariate Point Process + Poisson Regression Full model of Intensity Function Learning and Prediction Personalization Experiments Conclusion
23. Learning and Prediction Learning Prediction Given a topic keyword and tq, 1. Gather covariates info for tq. 1. Figure out covariates for intensity function For each visual cluster (VC) i, 2. Compute intensity function for each VC i, 2. Observe the actual occurrence of VC i 3. Sample L images according to M: No. of VCs J: No. of covariates T: No. of time steps O(MJ), for each tq O(MJT), only once offline 3. Compute MLE solution by using cyclic coordinate descent. 30 min (with soccer topic of 810K images) << 1 sec M: = 200, J = 118, T = 1,500
24. Outline Problem Statement Method Multivariate Point Process + Poisson Regression Full model of Intensity Function Learning and Prediction Personalization Experiments Conclusion
25. Personalization Idea of locally-weighted Learning [Atkeson et al.97] Collective Image prediction Each image is equally weighted For a user u6 Personalized Image prediction Each image is weighted according to the user similarity with u6 Learning is more biased.
26. Outline Problem Statement Method Multivariate Point Process + Poisson Regression Full model of Intensity Function Learning and Prediction Personalization Experiments Conclusion
27. Flickr Dataset 10,284,945 images of 40 topic keywords 7 groups Nations Places Animals Objects Activities Abstract Hot topics Ex. Soccer dataset Seasonal variation Zipf’s law
28. Experimental Tasks Split the dataset into training/test sets Randomly pick tq Training data + image DB 2010 Timeline 12/31/2008 ±1 days Randomly chosen 20 tq per topic Collective Image prediction L Predicted images Positive test images Randomly chosen 20 (tq,uq) pairs Personalized Image prediction
29. Evaluation Measures Actual images and predicted images are more then hundreds. How can we compare them? (1) Two distance metrics : Lower is better L2 Tiny [Torralbaet al. 2008] 2 2 * ** * ** SIFT/HOG Resize 32x32 images (2) Average precision: higher is better. Rank positive/negative test images Using predicted images
30. Quantitative Results Baselines Generative topic model Semantic meaning only State-of-the-art retrieval Collective Image prediction Personalized Image prediction Sampling from ImageNet Author-Time topic model PageRankbased IR 7~8% higher than the best baseline. 30
31. Examples of Collective Image Prediction World+cup Cardinals Ski+skating (a) Jan. Football / Snow (a) Jan. (b) May Bicycle+kayak+soccer (b) May Baseball / Leafy, Eggs (c) Sep. Soccer world cup (c) Sep. Baseball / Leafy
32. Examples of Personalized Image Prediction Fine+art Brazilian Painting (a) User1 Flower (a) User1 (b) User2 Class (b) User2 Dance (c) User3 Photography (c) User3 Auto-racing
33. Outline Problem Statement Method Multivariate Point Process + Poisson Regression Full model of Intensity Function Learning and Prediction Personalization Experiments Conclusion
34. Conclusion What’s done Web image prediction (1) User behavior prediction (2) Time-sensitive image reranking Poisson regression on multivariate point process Observations Example code will be available ! Many topics are associated with predictable periodic events. Image-based Personalization is important. More delicate information about user preference over texts Ex. What styles of painting does user A like?