1 / 30

Bayesian Co-clustering for Dyadic Data Analysis

This paper presents a Bayesian approach for co-clustering dyadic data, which captures relationships between two entities across various contexts, such as users and movies or genes and experiments. By employing mixed membership models, the method enables simultaneous clustering of both dimensions while accommodating sparsity in the data. The research demonstrates effective applications in gene expression analysis and collaborative filtering, showcasing the flexibility and stability of the Bayesian framework in matrix approximation tasks. Future work includes exploring joint decoding and predictive modeling.

myles-scott
Télécharger la présentation

Bayesian Co-clustering for Dyadic Data Analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Bayesian Co-clustering for Dyadic Data Analysis Arindam Banerjee banerjee@cs.umn.edu Dept of Computer Science & Engineering University of Minnesota, Twin Cities Workshop on Algorithms for Modern Massive Datasets (MMDS 2008) Joint work with Hanhuai Shan

  2. Introduction • Dyadic Data • Relationship between two entities • Examples • (Users, Movies): Ratings, Tags, Reviews • (Genes, Experiments): Expression • (Buyers, Products): Purchase, Ratings, Reviews • (Webpages, Advertisements): Click-through rate • Co-clustering • Simultaneous clustering of rows and columns • Matrix approximation based on co-clusters • Mixed membership co-clustering • Row/column has memberships in multiple row/column clusters • Flexible model, naturally handles sparsity Bayesian Co-clustering

  3. Example: Gene Expression Analysis Original Co-clustered Bayesian Co-clustering

  4. Co-clustering and Matrix Approximation Bayesian Co-clustering

  5. Example: Collaborative Filtering Bayesian Co-clustering

  6. Related Work • Partitional co-clustering • Bi-clustering (Hartigan ’72) • Bi-clustering of expression data (Cheng et al., ’00) • Information theoretic co-clustering (Dhillon et al., ’03) • Bregman co-clustering and matrix approximation (Banerjee et al., ’07) • Mixed membership models • Probabilistic latent semantic indexing (Hoffman, ’99) • Latent Dirichlet allocation (Blei et al., ’03) • Bayesian relational models • Stochastic block structure (Nowicki et al, ’01) • Infinite relational model (Kemp et al, ’06) • Mixed membership stochastic block model (Airoldi et al, ’07) Bayesian Co-clustering

  7. X2 X1 X3 X4 X5 Background • Bayesian Networks • Plates Bayesian Co-clustering

  8. document1 document2 Latent Dirichlet Allocation (LDA) [BNJ’03] Bayesian Co-clustering

  9. Bayesian Naïve Bayes (BNB) [BS’07] Bayesian Co-clustering

  10. Bayesian Co-clustering (BCC) Bayesian Co-clustering

  11. Bayesian Co-clustering (BCC) Bayesian Co-clustering

  12. Variational Inference • Expectation Maximization • Variational EM • Introduce a variational distribution to approximate . • Use Jensen’s inequality to get a tractable lower boundfor log-likelihood • Maximize the lower bound w.r.t for the best lower bound, i.e., minimize the KL divergence between and • Maximize the lower bound w.r.t Bayesian Co-clustering

  13. Variational Distribution • for each row, for each column Bayesian Co-clustering

  14. Variational EM for Bayesian Co-clustering = lower bound of log -likelihood Bayesian Co-clustering

  15. EM for Bayesian Co-clustering • Inference (E-step) • Parameter Estimation (M-step) (Gaussians) Bayesian Co-clustering

  16. Fast Latent Dirichlet Allocation (FastLDA) • Introduce a different variational distribution as an approximation of . • Number of variational parameters φ: m*n →n. • Number of optimizations over φ: m*n →n. FastLDA Original Bayesian Co-clustering

  17. FastLDA vs LDA: Perplexity Bayesian Co-clustering

  18. FastLDA vs LDA: Time Bayesian Co-clustering

  19. Word List for Topics (Classic3) LDA Fast LDA Bayesian Co-clustering

  20. Word List for Topics (Newsgroups) LDA Fast LDA Bayesian Co-clustering

  21. BCC Results: Simulated Data Bayesian Co-clustering

  22. BCC Results: Real Data • Movielens: Movie recommendation data • 100,000 ratings (1-5) for 1682 movies from 943 users (6.3%) • Binarize: 0 (1-3), 1(4-5). • Discrete (original), Bernoulli (binary) • Foodmart: Transaction data • 164,558 sales records for 7803 customers and 1559 products (1.35%) • Binarize: 0 (less than median), 1(higher than median) • Poisson (original), Bernoulli (binary) • Jester: Joke rating data • 100,000 ratings (-10.00 - +10.00) for 100 jokes from 1000 users (100%) • Binarize: 0 (lower than 0), 1 (higher than 0) • Gaussian (original), Bernoulli (binary) Bayesian Co-clustering

  23. BCC vs BNB vs LDA (Binary data) Training Set Test Set Perplexity on Binary Jester Dataset with Different Number of User Clusters Bayesian Co-clustering

  24. BCC vs BNB (Original data) Training Set Test Set Perplexity on Movielens Dataset with Different Number of User Clusters Bayesian Co-clustering

  25. Perplexity Comparison with 10 User Clusters Training Set Test Set On Binary Data Training Set Test Set On Original Data Bayesian Co-clustering

  26. Co-cluster Parameters (Movielens) Bayesian Co-clustering

  27. Co-embedding: Users Bayesian Co-clustering

  28. Co-embedding: Movies Bayesian Co-clustering

  29. Summary • Bayesian co-clustering • Mixed membership co-clustering for dyadic data • Flexible Bayesian priors over memberships • Applicable to variety of data types • Stable performance, consistently better in test set • Fast variational inference algorithm • One variational parameter for each row/column • Maintains coupling between row/column cluster memberships • Same idea leads to FastLDA (try it at home) • Future work • Open problem: Joint decoding of missing entries • Predictive models based on mixed membership co-clusters • Multi-relational clustering Bayesian Co-clustering

  30. References • A Generalized Maximum Entropy Approach to Bregman Co-clustering and Matrix ApproximationA. Banerjee, I. Dhillon, J. Ghosh, S. Merugu, D. Modha.Journal of Machine Learning Research (JMLR), (2007) . • Latent Dirichlet Conditional Naive Bayes ModelsA. Banerjee and H. Shan. IEEE International Conference on Data Mining (ICDM), (2007). • Latent Dirichlet AllocationD. Blei, A. Ng, M. Jordan.Journal of Machine Learning Research (JMLR), (2003). • Bayesian Co-clusteringH. Shan, A. Banerjee. Tech Report, University of Minnesota, Twin Cities, (2008). Bayesian Co-clustering

More Related