Physical Fluctuomatics Applied Stochastic Process 6th Graphical model and physical model

1. Physical FluctuomaticsApplied Stochastic Process 6th Graphical model and physical model Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University kazu@smapip.is.tohoku.ac.jp http://www.smapip.is.tohoku.ac.jp/~kazu/ Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

2. Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 5. Textbooks References • H. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001. • H. Nishimori, G. Ortiz: Elements of Phase Transitions and Critical Phenomena, Oxford University Press, 2011. • M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2010. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

3. Data is constructed from many bits • 010011101110101000111110000110000101000000111010101110101010 • 101101110001 ０,1 Bit Data A sequence is formed by deciding the arrangement of bits. Materials are constructed from a lot of molecules. Molecule Material Molecules have interactions of each other. System of a lot of elements with mutual relationCommon Concept between Information Sciences and Physics Main Interests Information Processing: Data Physics: Material, Natural Phenomena A lot of elements have mutual relation of each other Some physical concepts in Physical models are useful for the design of computational models in probabilistic information processing. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

4. Why is a physical viewpoint effective in probabilistic information processing? Matrials are constructed from a lot of molecules. (1023 molecules exist in 1 mol.) Molecules have intermolecular forces of each other Theoretical physicists always have to treat such multiple summation. Development of Approximate Methods Probabilistic information processing is also usually reduced to multiple summations or integrations. Application of physical approximate methods to probabilistic information processing Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

5. Probabilistic Model for Ferromagnetic Materials Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

6. Probabilistic Model for Ferromagnetic Materials > = > Prior probability prefers to the configuration with the least number of red lines. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

7. More is different in Probabilistic Model for Ferromagnetic Materials Large p Small p Sampling by Markov Chain Monte Carlo method Disordered State Ordered State Critical Point (Large fluctuation) More is different. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

8. Fundamental Probabilistic Models for Magnetic Materials -1 +1 Since h is positive, the probablity of up spin is larger than the one of down spin． h：External Field Average Variance Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

9. Fundamental Probabilistic Models for Magnetic Materials -1 -1 +1 +1 J：Interaction -1 +1 +1 -1 Since J is positive, (a1,a2)=(+1,+1) and (-1,-1) have the largest probability． Average Variance Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

10. Fundamental Probabilistic Models for Magnetic Materials E：Set of All the neighbouring Pairs of Nodes h h J J Translational Symmetry Problem: Compute Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

11. Fundamental Probabilistic Models for Magnetic Materials h h J J Translational Symmetry Problem: Compute Spontaneous Magnetization Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

12. h Jm Jm i Jm Jm Mean Field Approximation for Ising Model We assume that the probability for configurations satisfying are large. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

13. Mean Field Approximation for Ising Model We assume that all random variablesaiare independent of each other, approximately. Fixed Point Equation of m Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

14. Fixed Point Equation and Iterative Method • Fixed Point Equation Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

15. Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

16. Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

17. Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

18. Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

19. Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

20. Fixed Point Equation and Iterative Method • Fixed Point Equation • Iterative Method Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

21. h Jm Jm i Jm Jm Marginal Probability Distribution in Mean Field Approximation Jm：Mean Field Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

22. h l h l l h Advanced Mean Field Method l：Effective Field Bethe Approximation J Fixed Point Equation for l Kikuchi Method (Cluster Variation Meth) Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

23. h h J J Average of Ising Model on Square Grid Graph • Mean Field Approximation • Bethe Approximation • Kikuchi Method (Cluster Variation Method) • Exact Solution （L. Onsager，C.N.Yang） Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

24. Model Representation in Statistical Physics Gibbs Distribution Partition Function Energy Function Free Energy Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

25. Gibbs Distribution and Free Energy • Gibbs Distribution • Free Energy Variational Principle of Free Energy Functional F[Q] under Normalization Condition for Q(a) Free Energy Functional of Trial Probability Distribution Q(a) Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

26. Explicit Derivation of Variantional Principle for Minimization of Free Energy Functional Normalization Condition Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

27. Kullback-Leibler Divergence and Free Energy Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

28. Minimization of Kullback-Leibler Divergence between Interpretation of Mean Field Approximation as Information Theory and Marginal Probability Distributions Qi(ai) are determined so as to minimize D[Q|P] Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

29. Interpretation of Mean Field Approximation as Information Theory h h J J Translational Symmetry Problem: Compute Magnetization Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

30. Kullback-Leibler Divergence in Mean Field Approximation for Ising Model Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

31. Minimization of Kullback-Leibler Divergence and Mean Field Equation Set of all the neighbouring nodes of the node i Variation i Fixed Point Equations for {Qi|iV} Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

32. Orthogonal Functional Representation of Marginal Probability Distribution of Ising Model Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

33. Conventional Mean Field Equation in Ising Model h h J J Translational Symmetry Fixed Point Equation Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

34. Interpretation of Bethe Approximation (1) Translational Symmetry h h J J Compute and Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

35. Interpretation of Bethe Approximation (2) Free Energy KL Divergence Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

36. Free Energy KL Divergence Interpretation of Bethe Approximation (3) Bethe Free Energy Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

37. Interpretation of Bethe Approximation (4) Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

38. Interpretation of Bethe Approximation (5) Lagrange Multipliers to ensure the constraints Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

39. Interpretation of Bethe Approximation (6) • Extremum Condition Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

40. Interpretation of Bethe Approximation (7) Extremum Condition Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

41. 3 3 8 1 4 2 1 2 4 7 5 5 6 Interpretation of Bethe Approximation (8) Extremum Condition Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

42. 3 1 4 2 5 Interpretation of Bethe Approximation (9) Message Update Rule 3 8 1 2 4 7 5 6 Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

43. 3 8 3 1 2 4 7 5 6 1 4 2 3 5 1 4 2 5 Interpretation of Bethe Approximation (10) Message Passing Rule of Belief Propagation = It corresponds to Bethe approximation in the statistical mechanics. Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

44. Interpretation of Bethe Approximation (11) Translational Symmetry Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)

45. Summary • Statistical Physics and Information Theory • Probabilistic Model of Ferromagnetism • Mean Field Theory • Gibbs Distribution and Free Energy • Free Energy and Kullback-Leibler Divergence • Interpretation of Mean Field Approximation as Information Theory • Interpretation of Bethe Approximation as Information Theory Physics Fluctuomatics / Applied Stochastic Process (Tohoku University)