1 / 129

Graphical Models for the Internet

Graphical Models for the Internet. Amr Ahmed and Alexander Smola Yahoo Research, Santa Clara, CA. Thus far . Motivation Basic tools Clustering Topic Models Distributed batch inference Local and global states Star synchronization. Up next. Inference Online Distributed Sampling

candy
Télécharger la présentation

Graphical Models for the Internet

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Graphical Models for the Internet Amr Ahmed and Alexander Smola Yahoo Research, Santa Clara, CA

  2. Thus far ... • Motivation • Basic tools • Clustering • Topic Models • Distributed batch inference • Local and global states • Star synchronization

  3. Up next • Inference • Online Distributed Sampling • Single machine multi-threaded inference • Online EM and Submodular Selection • Applications • User tracking for behavioral Targeting • Content understanding • User modeling for content recommendation

  4. 4. Online Model

  5. Scenarios Time Time • Batch Large-Scale • Covered in part 1 • Mini-batches • We already have a model • Data arrives in batches • We would like to keep model up-to-data • Time-sensitive • Data arrives one item at a time • Model should be up-to-data

  6. 4.1 Dynamic Clustering

  7. The Chinese Restaurant Process • Allows the number of mixtures to grow with the data • They are called non-parametric models • Means the number of effective parameters grow with data • Still have hyper-parametersthat control the rate of growth • a:how fast a new cluster/mixture is born? • G0: Prior over mixture component parameters

  8. The Chinese Restaurant Process f1 f2 f3 Generative Process • For data point xi • Choose table j mj and Sample xi ~ f(fj) • Choose a new table K+1  a • Sample fK+1 ~ G0 and Sample xi ~ f(fK+1) The rich gets richer effect CANNOT handle sequential data

  9. Recurrent CRP (RCRP) [Ahmed and Xing 2008] • Adapts the number of mixture components over time • Mixture components can die out • New mixture components are born at any time • Retained mixture components parametersevolve according to a Markovian dynamics

  10. The Recurrent Chinese Restaurant Process T=1 Dish eaten at table 3 at time epoch 1 OR the parameters of cluster 3 at time epoch 1 f1,1 f2,1 f3,1 Generative Process • Customers at time T=1 are seated as before: • Choose table j mj,1 and Sample xi ~ f(fj,1) • Choose a new table K+1 a • Sample fK+1,1 ~ G0 and Sample xi ~ f(fK+1,1)

  11. The Recurrent Chinese Restaurant Process f1,1 f1,1 f2,1 f2,1 f3,1 f3,1 T=1 m'2,1=3 m'3,1=1 m'1,1=2 T=2

  12. T=1 f1,1 f2,1 f3,1 f1,1 f2,1 f3,1 T=2 m'2,1=3 m'3,1=1 m'1,1=2

  13. T=1 f1,1 f2,1 f3,1 f1,1 f2,1 f3,1 T=2 m'2,1=3 m'3,1=1 m'1,1=2

  14. T=1 f1,1 f2,1 f3,1 f1,1 f2,1 f3,1 T=2 m'2,1=3 m'3,1=1 m'1,1=2

  15. T=1 f1,2 f2,1 f3,1 f1,1 f2,1 f3,1 T=2 m'2,1=3 m'3,1=1 m'1,1=2 Sample f1,2 ~ P(.| f1,1)

  16. T=1 f1,2 f2,1 f3,1 f1,1 f2,1 f3,1 T=2 m'2,1=3 m'3,1=1 m'1,1=2 And so on ……

  17. T=1 f1,2 f2,2 f3,1 f1,1 f2,1 f3,1 f4,2 T=2 m'2,1=3 m'3,1=1 m'1,1=2 Died out cluster Newly born cluster At the end of epoch 2

  18. T=1 f1,1 f2,1 f3,1 f1,2 f1,2 f2,2 f2,2 f3,1 f4,2 f4,2 T=2 N2,1=3 m'3,1=1 N1,1=2 m'2,2=2 m'1,2=2 m'4,2=1 T=3

  19. Recurrent Chinese Restaurant Process æ ö - h H å ç ÷ e m r - k , t h è ø = h 1 • Can be extended to model higher-order dependencies • Can decay dependencies over time • Pseudo-counts for table k at time t is History size Number of customers sitting at table K at time epoch t-h Decay factory

  20. T=1 f1,1 f2,1 f3,1 f1,2 f2,2 f3,1 f4,2 f1,2 f2,2 f4,2 T=2 m'2,1=3 m'3,1=1 m'1,1=2 m'2,3 T=3 æ ö - h H å ç ÷ e m r m'2,3 = - k , t h è ø = h 1

  21. 4.2 Online Distributed Inference Tracking Users Interest

  22. Characterizing User Interests • Short term vs long-term Music Housing Buying a car Furniture Travel plans Jan July Oct April

  23. Characterizing User Interests • Short term vs long-term • Latent Jan July Oct April Gaga mortgage millage Barcelona fast seafood used

  24. Problem formulation Input • Queries issued by the user or tags of watched content • Snippet of page examined by user • Time stamp of each action (day resolution) Output • Users’ daily distribution over interests • Dynamic interest representation • Online and scalable inference • Language independent Flight London Hotel weather School Supplies Loan semester classes registration housing rent

  25. Problem formulation Input • Queries issued by the user or tags of watched content • Snippet of page examined by user • Time stamp of each action (day resolution) Output • Users’ daily distribution over interests • Dynamic interest representation • Online and scalable inference • Language independent Back To school finance Travel Flight London Hotel weather School Supplies Loan semester classes registration housing rent

  26. Problem formulation When to show a financing ad? Back To school finance Travel Flight London Hotel weather School Supplies Loan semester classes registration housing rent

  27. Problem formulation When to show a financing ad? Back To school finance Travel Flight London Hotel weather School Supplies Loan semester classes registration housing rent

  28. Problem formulation When to show a financing ad? Back To school finance Travel Flight London Hotel weather School Supplies Loan semester classes registration housing rent

  29. Problem formulation When to show a hotel ad? Back To school finance Travel Flight London Hotel weather School Supplies Loan semester classes registration housing rent

  30. Problem formulation When to show a hotel ad? Back To school finance Travel Flight London Hotel weather School Supplies Loan semester classes registration housing rent

  31. Problem formulation Input • Queries issued by the user or tags of watched content • Snippet of page examined by user • Time stamp of each action (day resolution) Output • Users’ daily distribution over interests • Dynamic interest representation • Online and scalable inference • Language independent Back To school finance Travel Flight London Hotel weather School Supplies Loan semester classes registration housing rent

  32. Mixed-Membership Formulation Job Hiring speed price part-timeCamry Career opening bonus package carddiet calories loanrecipe milk Weight lb kg Objects Degree of membership Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Blue Book Kelley Prices Small Speed large job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase Mixtures Diet Job Cars Finance

  33. In Graphical Notation

  34. In Polya-Urn Representation • Collapse multinomial variables: • Fixed-dimensional Hierarchal Polya-Urn representation • Chinese restaurant franchise x x

  35. Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Blue Book Kelley Prices Small Speed large job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase Global topics trends Food Chicken Topic word-distributions User-specific topics trends (mixing-vector) Car speed offer camryaccordcareer User interactions: queries, keyword from pages viewed

  36. Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Blue Book Kelley Prices Small Speed large job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase Food Chicken ……… Generative Process Car speed offer camryaccordcareer • For each user interaction • Choose an intent from local distribution • Sample word from the topic’s word-distribution • Choose a new intent  l • Sample a new intent from the global distribution • Sample word from the new topic word-distribution

  37. Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Blue Book Kelley Prices Small Speed large job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase Food Chicken pizza ……… Generative Process Car speed offer camryaccordcareer • For each user interaction • Choose an intent from local distribution • Sample word from the topic’s word-distribution • Choose a new intent  l • Sample a new intent from the global distribution • Sample word from the new topic word-distribution

  38. Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Blue Book Kelley Prices Small Speed large job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase Food Chicken pizza ……… Generative Process Car speed offer camryaccordcareer • For each user interaction • Choose an intent from local distribution • Sample word from the topic’s word-distribution • Choose a new intent  l • Sample a new intent from the global distribution • Sample word from the new topic word-distribution

  39. Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Blue Book Kelley Prices Small Speed large job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase Food Chicken pizza hiring ……… Generative Process Car speed offer camryaccordcareer • For each user interaction • Choose an intent from local distribution • Sample word from the topic’s word-distribution • Choose a new intent  l • Sample a new intent from the global distribution • Sample word from the new topic word-distribution

  40. Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Blue Book Kelley Prices Small Speed large job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase Food Chicken pizza millage Generative Process Car speed offer camryaccordcareer • For each user interaction • Choose an intent from local distribution • Sample word from topic’s word-distribution • Choose a new intent  l • Sample a new intent from the global distribution • Sample from word the new topic word-distribution

  41. Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Blue Book Kelley Prices Small Speed large job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase Food Chicken pizza millage Car speed offer camryaccordcareer Problems • Static Model • Does not evolve user’s interests • Does not evolve the global trend of interests • Does not evolve interest’s distribution over terms

  42. At time t At time t+1 Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Blue Book Kelley Prices Small Speed large job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase Food Chicken pizza millage Build a dynamic model Connect each level using a RCRP Car speed offer camryaccordcareer

  43. At time t At time t+1 At time t+2 At time t+3 Global process m m' n User 1 process n' Which time kernel to use at each level? User 2 process User 3 process

  44. At time t At time t+1 Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Blue Book Kelley Prices Small Speed large job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase Food Chicken pizza millage = * Pseudo counts Decay factor Car speed offer camryaccordcareer Observation 1 • Popular topics at time t are likely to be popular at time t+1 • fk,t+1 is likely to smoothly evolve from fk,t

  45. At time t At time t+1 Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Blue Book Kelley Prices Small Speed large Car Blue Book Kelley Prices Small Speed large job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase Food Chicken pizza millage Car Altima Accord Book Kelley Prices Small Speed Intuition Captures current trend of the car industry (new release for e.g.) ~ Car speed offer camryaccordcareer fk,t fk,t+1 ~ Dir(bk,t+1) Observation 1 • Popular topics at time t are likely to be popular at time t+1 • fk,t+1is likely to smoothly evolve from fk,t

  46. At time t At time t+1 Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Altima Accord Blue Book Kelley Prices Small Speed job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase Food Chicken pizza millage How do we get a prior that captures both long and short term interest? Car speed offer camryaccordcareer Observation 2 • User prior at time t+1 is a mixture of the user short and long term interest

  47. All μ3 month μ2 week Long-term μ short-term Prior for user actions at time t food chicken Pizza millage Food Chicken pizza Part-time Opening salary Kelly recipe cuisine recipe job hiring t t+1 Time Diet Job Cars Finance Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Blue Book Kelley Prices Small Speed large job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase

  48. At time t At time t+1 Recipe Chocolate Pizza Food Chicken Milk Butter Powder Car Altima Accord Blue Book Kelley Prices Small Speed job Career Business Assistant Hiring Part-time Receptionist Bank Online Credit Card debt portfolio Finance Chase Food Chicken Pizza millage short-term priors Generative Process Car speed offer camryaccordcareer • For each user interaction • Choose an intent from local distribution • Sample word from the topic’s word-distribution • Choose a new intent  l • Sample a new intent from the global distribution • Sample word from the new topic word-distribution

  49. Polya-Urn RCRF Process ?

  50. Simplified Graphical Model ~ ~ ~ ~ At time t At time t+1 ~ ~

More Related