Create Presentation
Download Presentation

Download Presentation

Tokyo Institute of Technology, Japan Yu Nishiyama and Sumio Watanabe

Tokyo Institute of Technology, Japan Yu Nishiyama and Sumio Watanabe

115 Views

Download Presentation
Download Presentation
## Tokyo Institute of Technology, Japan Yu Nishiyama and Sumio Watanabe

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Tokyo Institute of Technology, Japan**Yu Nishiyama and Sumio Watanabe Theoretical Analysis of Accuracy of Gaussian Belief Propagation**Background**Belief propagation (BP) The algorithm which computes marginal distributions efficiently Marginal distribution: requires huge computational cost.**Variety of Research Areas**ex. (i) Probabilistic inference for AI (ii) Error correcting code (LDPC, Turbo codes) noise correcting 000101 000111 000101 (iii) Code division multiple access (CDMA) (iv) Probabilistic image processing restore degrade image restored image**Properties of BP & Loopy BP (LBP)**Tree-structured target distribution Exact marginal probabilities Loop-structured target distribution Convergence? Approximate marginal probabilities Ex. T. Heskes, ”On the Uniqueness of Loopy Belief Propagation Fixed Points”, Neural Computation16(11), 2379-2414, 2004. Y. Weiss,”Correctness of belief propagation in graphical models with arbitrary topology”, Neural Computation13(10), 2173-2200, 2001.**Purpose**We analytically clarify the accuracy of LBP when the target distribution is a Gaussian distribution. What is the conditions for LBP convergence? How close is the LBP solution to the true marginal distributions? In Probabilistic image processing K. Tanaka, H. Shouno, M. Okada, “Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing”, J.phys. A, Math. Gen., vol.37, no.36, pp.8675-8696, 2004.**Table of Contents**・BP ＆ LBP ・Gaussian Distribution ・Main Results (i) Single Loop (ii) Graphs with Multi-loops ・Conclusion**Graphical Models**Target distribution Marginal distribution**BP ＆ LBP**Marginal distribution**If it converges**How are messages decided? are decided by the fixed-points Messages of a message update rule: a fixed-point**Gaussian Distribution**Target distribution: ( Inverse covariance matrix) Messages: Update rule:**Fixed-Points of Messages**Theorem1 When a Gaussian distribution forms a single loop, the fixed-points of messages are given by Single loop are the cofactors. where**LBP Solution**Theorem 2 The solution of LBP is given by where**True**Loopy Belief Propagation Belief Propagation Intuitive Understanding LBP Solution Loop Tree**Accuracy of LBP**Theorem 3 The Kullback-Leibler (KL) distances are calculated as True marginal density Solution of LBP where is given by Convergence condition is since**Graphs with Multi-Loops**How about the graphs having arbitrary structures? We clarify the LBP solution at small covariances. Multi-loops We derive the expansions w. r. t. where inverse covariance matrix is**A Fixed-Point of Inverse Variances**Theorem 4 A fixed-point of inverse variances satisfies the following system of equations: The solution of the system is expanded as**Comparison with true inverse variances**Expansions of LBP solution are True inverse variances are**Accuracy of LBP**Theorem 5 The Kullback-Leibler (KL) distances are expanded as True marginal density Solution of LBP where are**Conclusion**(i) For a single loop, we revealed the parameter that determines the accuracy of LBP and the condition that tells us when LBP converges. (ii) For arbitrary structures, we revealed the expansions of LBP solution at small covariances and the accuracy. These fundamental results contribute to understanding the theoretical properties underlying LBP. We analytically clarified the accuracy of LBP in a Gaussian distribution.