Tokyo Institute of Technology, Japan Yu Nishiyama and Sumio Watanabe Theoretical Analysis of Accuracy of Gaussian Belief Propagation
Background Belief propagation (BP) The algorithm which computes marginal distributions efficiently Marginal distribution: requires huge computational cost.
Variety of Research Areas ex. (i) Probabilistic inference for AI (ii) Error correcting code (LDPC, Turbo codes) noise correcting 000101 000111 000101 (iii) Code division multiple access (CDMA) (iv) Probabilistic image processing restore degrade image restored image
Properties of BP & Loopy BP (LBP) Tree-structured target distribution Exact marginal probabilities Loop-structured target distribution Convergence? Approximate marginal probabilities Ex. T. Heskes, ”On the Uniqueness of Loopy Belief Propagation Fixed Points”, Neural Computation16(11), 2379-2414, 2004. Y. Weiss,”Correctness of belief propagation in graphical models with arbitrary topology”, Neural Computation13(10), 2173-2200, 2001.
Purpose We analytically clarify the accuracy of LBP when the target distribution is a Gaussian distribution. What is the conditions for LBP convergence? How close is the LBP solution to the true marginal distributions? In Probabilistic image processing K. Tanaka, H. Shouno, M. Okada, “Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing”, J.phys. A, Math. Gen., vol.37, no.36, pp.8675-8696, 2004.
Table of Contents ・BP ＆ LBP ・Gaussian Distribution ・Main Results (i) Single Loop (ii) Graphs with Multi-loops ・Conclusion
Graphical Models Target distribution Marginal distribution
BP ＆ LBP Marginal distribution
If it converges How are messages decided? are decided by the fixed-points Messages of a message update rule: a fixed-point
Gaussian Distribution Target distribution: ( Inverse covariance matrix) Messages: Update rule:
Fixed-Points of Messages Theorem1 When a Gaussian distribution forms a single loop, the fixed-points of messages are given by Single loop are the cofactors. where
LBP Solution Theorem 2 The solution of LBP is given by where
True Loopy Belief Propagation Belief Propagation Intuitive Understanding LBP Solution Loop Tree
Accuracy of LBP Theorem 3 The Kullback-Leibler (KL) distances are calculated as True marginal density Solution of LBP where is given by Convergence condition is since
Graphs with Multi-Loops How about the graphs having arbitrary structures? We clarify the LBP solution at small covariances. Multi-loops We derive the expansions w. r. t. where inverse covariance matrix is
A Fixed-Point of Inverse Variances Theorem 4 A fixed-point of inverse variances satisfies the following system of equations: The solution of the system is expanded as
Comparison with true inverse variances Expansions of LBP solution are True inverse variances are
Accuracy of LBP Theorem 5 The Kullback-Leibler (KL) distances are expanded as True marginal density Solution of LBP where are
Conclusion (i) For a single loop, we revealed the parameter that determines the accuracy of LBP and the condition that tells us when LBP converges. (ii) For arbitrary structures, we revealed the expansions of LBP solution at small covariances and the accuracy. These fundamental results contribute to understanding the theoretical properties underlying LBP. We analytically clarified the accuracy of LBP in a Gaussian distribution.