Maximizing multi-information. (English) Zbl 1249.82011

Summary: Stochastic interdependence of a probability distribution on a product space is measured by its Kullback-Leibler distance from the exponential family of product distributions (called multi-information). Here, we investigate low-dimensional exponential families that contain the maximizers of stochastic interdependence in their closure.
Based on a detailed description of the structure of probability distributions with globally maximal multi-information, we obtain our main result: the exponential family of pure pair interactions contains all global maximizers of the multi-information in its closure.


82C32 Neural nets applied to problems in time-dependent statistical mechanics
62B10 Statistical aspects of information-theoretic topics
94A15 Information theory (general)
68T05 Learning and adaptive systems in artificial intelligence
Full Text: arXiv EuDML Link


[1] Aarts E., Korst J.: Simulated Annealing and Boltzmann Machines. Wiley, New York 1989 · Zbl 0674.90059
[2] Ackley D. H., Hinton G. E., Sejnowski T. J.: A learning algorithm for Boltzmann machines. Cognitive Science 9 (1985), 147-169 · doi:10.1207/s15516709cog0901_7
[3] Aigner M.: Combinatorial Theory, Classics in Mathematics. Springer-Verlag, Berlin 1997
[4] Amari S.: Information geometry on hierarchy of probability distributions. IEEE Trans. Inform. Theory 47 (2001), 1701-1711 · Zbl 0997.94009 · doi:10.1109/18.930911
[5] Amari S., Kurata, K., Nagaoka H.: Information geometry of Boltzmann machines. IEEE Trans. Neural Networks 3 (1992), 2, 260-271 · doi:10.1109/72.125867
[6] Ay N.: An information-geometric approach to a theory of pragmatic structuring. Ann. Probab. 30 (2002), 416-436 · Zbl 1010.62007 · doi:10.1214/aop/1020107773
[7] Ay N.: Locality of global stochastic interaction in directed acyclic networks. Neural Computation 14 (2002), 2959-2980 · Zbl 1079.68582 · doi:10.1162/089976602760805368
[8] Linsker R.: Self-organization in a perceptual network. IEEE Computer 21 (1988), 105-117 · doi:10.1109/2.36
[9] Matúš F., Ay N.: On maximization of the information divergence from an exponential family. Proc. WUPES’03 (J. Vejnarová, University of Economics, Prague 2003, pp. 199-204
[10] Shannon C. E.: A mathematical theory of communication. Bell System Tech. J. 27 (1948), 379-423, 623-656 · Zbl 1154.94303 · doi:10.1002/j.1538-7305.1948.tb01338.x
[11] Tononi G., Sporns, O., Edelman G. M.: A measure for brain complexity: Relating functional segregation and integration in the nervous system. Proc. Nat. Acad. Sci. U. S. A. 91 (1994), 5033-5037 · doi:10.1073/pnas.91.11.5033
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.