Maximizing multi-information. (English) Zbl 1249.82011

Ay, Nihat; Knauf, Andreas

Kybernetika 42, No. 5, 517-538 (2006).

Summary: Stochastic interdependence of a probability distribution on a product space is measured by its Kullback-Leibler distance from the exponential family of product distributions (called multi-information). Here, we investigate low-dimensional exponential families that contain the maximizers of stochastic interdependence in their closure.
Based on a detailed description of the structure of probability distributions with globally maximal multi-information, we obtain our main result: the exponential family of pure pair interactions contains all global maximizers of the multi-information in its closure.

MSC:

82C32	Neural nets applied to problems in time-dependent statistical mechanics
62B10	Statistical aspects of information-theoretic topics
94A15	Information theory (general)
68T05	Learning and adaptive systems in artificial intelligence

Keywords:

multi-information; relative entropy; pair-interaction; infomax principle; Boltzmann machine; neural networks

References:

[1]	Aarts E., Korst J.: Simulated Annealing and Boltzmann Machines. Wiley, New York 1989 · Zbl 0674.90059
[2]	Ackley D. H., Hinton G. E., Sejnowski T. J.: A learning algorithm for Boltzmann machines. Cognitive Science 9 (1985), 147-169
[3]	Aigner M.: Combinatorial Theory, Classics in Mathematics. Springer-Verlag, Berlin 1997
[4]	Amari S.: Information geometry on hierarchy of probability distributions. IEEE Trans. Inform. Theory 47 (2001), 1701-1711 · Zbl 0997.94009
[5]	Amari S., Kurata, K., Nagaoka H.: Information geometry of Boltzmann machines. IEEE Trans. Neural Networks 3 (1992), 2, 260-271
[6]	Ay N.: An information-geometric approach to a theory of pragmatic structuring. Ann. Probab. 30 (2002), 416-436 · Zbl 1010.62007
[7]	Ay N.: Locality of global stochastic interaction in directed acyclic networks. Neural Computation 14 (2002), 2959-2980 · Zbl 1079.68582
[8]	Linsker R.: Self-organization in a perceptual network. IEEE Computer 21 (1988), 105-117 · Zbl 05087941
[9]	Matúš F., Ay N.: On maximization of the information divergence from an exponential family. Proc. WUPES’03 (J. Vejnarová, University of Economics, Prague 2003, pp. 199-204
[10]	Shannon C. E.: A mathematical theory of communication. Bell System Tech. J. 27 (1948), 379-423, 623-656 · Zbl 1154.94303
[11]	Tononi G., Sporns, O., Edelman G. M.: A measure for brain complexity: Relating functional segregation and integration in the nervous system. Proc. Nat. Acad. Sci. U. S. A. 91 (1994), 5033-5037