Create Presentation
Download Presentation

Download Presentation
## Information Geometry of MaxEnt Principle

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**MaxEnt 07’**Information Geometry of MaxEnt Principle Shun-ichi Amari RIKEN Brain Science Institute**Information Geometry**Systems Theory Information Theory Statistics Neural Networks Combinatorics Physics Information Sciences Math. AI Riemannian Manifold Dual Affine Connections Manifold of Probability Distributions**Information Geometry ?**Riemannian metric Dual affine connections**Invariance**1. Invariant under reparameterization 2. Invariant under different representation**Two Structures**Riemannian metric—Fisher information Affine connection -- geodesic, straight line how curved is the manifold?**Kullback-Leibler Divergence**quasi-distance**KL-divergence and RiemannianStructure**relation Fisher information matrix**AffineConnection**covariant derivative straight line**Renyi-Tasallis**Exponential connection Entropy KL-divergence Mixture connection Levi-Civita (Riemannian)**Affine Connections**e-geodesic m-geodesic**Duality**Y X Y X Riemannian geometry:**Dually flat manifold**S = {p(x), x discrete}**Dually Flat Manifold**1. Potential Functions ---convex (Legendre transformation) 2. Divergence KL-divergence 3. Pythagoras Theorem 4. Projection Theorem**Projection Theorem**m-geodesic e-geodesic**Applications to Statistics**curved exponential family: : estimation : testing**High-Order Asymptotics**:Cramér-Rao**Other Applications**• Systems theory • Information theory • Neuromanifold • Belief propagation • Boosting (Murata-Eguchi) • Higher-order correlations • Mathematics --- Orlicz space (Pistone, Gracceli) • Physics --- Amari-Nagaoka, Methods of Information Geometry, AMS & Oxford U., 2000 Amari, Differential-Geometrical Methods of Statistics, Springer, 1985 Kass and Vos, Geomtrical Foundations of Asymptotic Inference, Wiley, 1997 Murrey and Rice, Differential Geometry and Statistics, Chapman, 1993**Exponential Family : dually flat**Two coordinate systems**Exponential Family**example (1) : discrete distributions Negative entropy**example (2) : Gaussian distributions**example (3) : AR model**Divergence**Pythagorean Theorem m-flat e-flat**Divergence and Entropy**equi-divergence: equi-entropy**Dual Foliation**Pythagorean theorem**Geometry**Potentials**Rényi-Tsallis entropy**Manifold of positive measures m(x)**Entropy (alpha-entropy) is a fundamental quantity---- It is**given rise to from a fundamental geometrical structure. KL-divergence is derived therefrom.