On the storage capacity of Hopfield models with correlated patterns. (English) Zbl 0941.60090

It is well known that the Hopfield model of a fully connected neural network with \(N\) neurons with the original Hebbian learning rule can store a number of patters equal to \(C N/\ln N\) without error, and equal to \(\alpha_c N\) (with \(\alpha_c\approx 0.138\) according to numerical and replica results) if a small fraction of errors is tolerated. These results hold with probability tending to one rapidly if the patterns are assumed to be i.i.d. random variables with each component a symmetric i.i.d. Rademacher random variable. The present paper presents a rigorous investigation or the question what becomes of these results if the patterns are correlated, either “semantically” or “spatially”. More specifically, it is supposed that either for any \(i\), \(\xi^\mu_i\), \(\mu\in {\mathbb N}\), form a Markov chain with state space \(\{-1,+1\}\), which are in turn independent for different \(i\), or vice versa that for each \(\mu\), \(\xi^\mu_i,i\in {\mathbb N}\), is an family of independent Markov chains. The results obtained are the following: 1) If perfect storage is required, the storage capacity decreases linearly the degree of correlation increases in the semantic case, while the opposite is true in the case of spatial correlation. 2) If storage with error is tolerated, the bounds obtained for \(a_c\) decrease with the correlation in both cases.
Reviewer: A.Bovier (Berlin)


60K35 Interacting random processes; statistical mechanics type models; percolation theory
82C32 Neural nets applied to problems in time-dependent statistical mechanics
Full Text: DOI


[1] Amit, D. J. (1987). The properties of models of simple neural networks. Heidelberg Colloquium on Glassy Dy namics. Lecture Notes in Phy s. 275. Springer, Berlin.
[2] Amit, D. J., Gutfreund, G. and Sompolinsky, H. (1985). Spin-glass models of neural networks. Phy s. Rev. A 32 1007-1018.
[3] Amit, D. J., Gutfreund, G. and Sompolinsky, H. (1987). Statistical mechanics of neural networks near saturation. Ann. physics 173 30-67.
[4] Bovier, A. and Gay rard, V. (1992). Rigorous bounds on the storage capacity of the dilute Hopfield model. J. Statist. Phy s. 69 597-627. · Zbl 0900.82064
[5] Bovier, A. and Gay rard, V. (1996). An almost sure large deviation principle for the Hopfield model. Ann. Probab. 24 1444-1475. · Zbl 0871.60022
[6] Bovier, A. and Gay rard, V. (1997). Hopfield models as a generalized mean field model. In Mathematics of Spin Glasses and Neural Networks (A. Bovier and P. Picco, eds.). Birkhäuser, Boston.
[7] Bovier, A. and Gay rard, V. (1997). The retrieval phase of the Hopfield model. Probab. Theory Related Fields 107 61-98. · Zbl 0866.60085
[8] Bovier, A., Gay rard, V. and Picco, P. (1994). Gibbs states for the Hopfield model in the regime of perfect memory. Probab. Theory Related Fields 100 329-363. · Zbl 0810.60094
[9] Bovier, A., Gay rard, V. and Picco, P. (1995). Large deviation principles for the Hopfield model and the Kac-Hopfield model. Probab. Theory Related Fields 101 511-546. · Zbl 0826.60090
[10] Bovier, A., Gay rard, V. and Picco, P. (1995). Gibbs states for the Hopfield model with extensively many patterns. J. Statist. Phy s. 79 395-414. · Zbl 1081.82570
[11] Drey fus, G., Guy on, I. and Personnaz, L. (1986). Neural network design for efficient infor mation retrieval. Disordered sy stems and biological organization (Les Houches, 1985). NATO Adv. Sci. Inst. Ser. F Comput. Sy stems Sci. 20 227-231. · Zbl 1356.82028
[12] Gentz, B. (1996). A central limit theorem for the overlap parameter in the Hopfield model. Ann. Probab. 24 1809-1841. · Zbl 0872.60015
[13] Georgii, H.-O. (1988). Gibbs measures and phase transition. In Studies in Mathematics 9 (H. V. Bauer, J. Heinz-Kazden and E. Zehnder, eds.). de Gruy ter, Berlin. · Zbl 0657.60122
[14] Hopfield, J. J. (1982). Neural networks and physical sy stems with emergent collective computational abilities. Proc. Nat. Acad. Sci. U.S.A. 79 2554-2558. JSTOR: · Zbl 1369.92007
[15] K ühn, R. and Steffan, H. (1994). Replica sy mmetry breaking in attractor neural network models. Z. Phy s. B 95 249-260.
[16] Loukianova, D. (1994). Capacité de mémoire dans le mod ele de Hopfield. C.R. Acad. Sci. Paris 318 157-160. · Zbl 0794.92002
[17] Loukianova, D. (1997). Lower bounds on the restitution error in the Hopfield model. Probab. Theory Related Fields 107 161-176. · Zbl 0870.60021
[18] McEliece, R., Posner, E., Rodemich, E. and Venkatesh, S. (1987). The capacity of the Hopfield associative memory. IEEE Trans. Inform. Theory 33 461-482. · Zbl 0631.94016
[19] Miy ashita, Y. (1988). Neuronal correlate of visual associative long term memory in the primate temporal cortex. Nature 335 817-819.
[20] Monasson, R. (1992). Properties of neural networks storing spatially correlated patterns. J. Phy s. A Math. Gen. 335 3701-3720. · Zbl 0791.68140
[21] Newman, C. (1988). Memory capacity in neural networks. Neural Networks 1 223-238.
[22] Pastur, L. A. and Figotin, A. L. (1977). Exactly soluble model of a spin-glass. Soviet J. of Low Temperature Phy s. 3 378-383.
[23] Petritis, D. (1995). Thermody namic formalism of neural computing. Univ. Rennes I.
[24] Sandmeier, M. (1997). On the storage capacity of neural networks with temporal association. Ph.D. thesis. Univ. Bielefeld. · Zbl 0889.92002
[25] Talagrand, M. (1995). Résultats rigoureux pour le mod ele de Hopfield. C.R. Acad. Sci. Paris Ser. I 321 309-312. · Zbl 0845.60102
[26] Talagrand, M. (1996). Rigorous results of the Hopfield model with many patterns. · Zbl 0897.60041
[27] Tarkowski, W. and Lewenstein, M. (1993). Storage of sets of correlated data in neural network memories. J. Phy s. A Math. Gen. 26 2453-2469. · Zbl 0788.68118
[28] van Hemmen, L. and K ühn, R. (1991). Collective phenomena in neural networks. In Models of Neural Networks (E. Domany, L. v. Hemmen and R. Schulte, eds.). Springer, Berlin. · Zbl 0783.92006
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.