×

zbMATH — the first resource for mathematics

Storing object-dependent sparse codes in a Willshaw associative network. (English) Zbl 07268865
Summary: Willshaw networks are single-layered neural networks that store associations between binary vectors. Using only binary weights, these networks can be implemented efficiently to store large numbers of patterns and allow for fault-tolerant recovery of those patterns from noisy cues. However, this is only the case when the involved codes are sparse and randomly generated. In this letter, we use a recently proposed approach that maps visual patterns into informative binary features. By doing so, we manage to transform MNIST handwritten digits into well-distributed codes that we then store in a Willshaw network in autoassociation. We perform experiments with both noisy and noiseless cues and verify a tenuous impact on the recovered pattern’s relevant information. More specifically, we were able to perform retrieval after filling the memory to several factors of its number of units while preserving the information of the class to which the pattern belongs.
MSC:
68 Computer science
Software:
MNIST
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Amari, S. (1989). Characteristics of sparsely encoded associative memory. Neural Networks, 2(6), 451-457. ,
[2] Amit, D. J., Gutfreund, H., & Sompolinsky, H. (1985). Spin-glass models of neural networks. Physical Review A, 32(2), 1007-1018. ,
[3] Barlow, H. B. (1972). Single units and sensation: A neuron doctrine for perceptual psychology? Perception, 1(4), 371-394. ,
[4] Chafee, M. V., Averbeck, B., & A. Crowe, D. (2007). Representing spatial relationships in posterior parietal cortex: Single neurons code object-referenced position. Cerebral Cortex, 17, 2914-2932. ,
[5] Ferro, D.,Gripon, V., & Jiang, X. (2016). Nearest neighbour search using binary neural networks. In Proceedings of the International Joint Conference on Neural Networks. Piscataway, NJ: IEEE. ,
[6] Field, D. J. (1994). What is the goal of sensory coding? Neural Computation, 6, 559-601. ,
[7] Gardner, E. (1988). The space of interactions in neural network models. Journal of Physics A: Mathematical and General, 21, 257-270. , · Zbl 1128.82302
[8] Gripon, V., M. Lowe, V., & Vermet, F. (2018). Associative memories to accelerate approximate nearest neighbor search. Applied Sciences, 8(9), 1676. ,
[9] Hebb, D. O. (1949). The organization of behavior: A neuropsychological theory. New York: Wiley-Interscience.
[10] Hecht-Nielsen, R. (1989). Neurocomputing. Reading, MA: Addison-Wesley.
[11] Knoblauch, A. (2003). Optimal matrix compression yields storage capacity 1 for binary Willshaw associative memory. In O. Kaynak, E. Alpaydin, E. Oja, & L. Xu (Eds.), Artificial neural networks and neural information processing, LNCS 2714, 325-332. · Zbl 1037.68668
[12] Knoblauch, A. (2008). Neural associative memory and the Willshaw-Palm probability distribution. SIAM Journal on Applied Mathematics, 69, 169-196. , · Zbl 1156.92003
[13] Knoblauch, A. (2010). Zip nets: Efficient associative computation with binary synapses. In Proceedings of the 2010 International Joint Conference on Neural Networks (pp. 4271-4278). Piscataway, NJ: IEEE. ,
[14] Knoblauch, A. (2011). Neural associative memory with optimal Bayesian learning. Neural Computation, 23, 1393-1451. , · Zbl 1216.68216
[15] Knoblauch, A. (2012). Method and device for realizing an associative memory based on inhibitory neural networks. U.S. Patent No. 8,335,752.
[16] Knoblauch, A. (2013). Method and device for realizing a neural associative memory based on nonlinear learning of discrete synapses. U.S. Patent No. 8,463,722.
[17] Knoblauch, A. (2016). Efficient associative computation with discrete synapses. Neural Computation, 28, 118-186. , · Zbl 07062518
[18] Knoblauch, A., Palm, G., & Sommer, F. T. (2010). Memory capacities for synaptic and structural plasticity. Neural Computation, 22, 289-341. , · Zbl 1183.92017
[19] Laughlin, S. B., & Sejnowski, T. J. (2003). Communication in neuronal networks. Science, 301(5641), 1870-1874. ,
[20] LeCun Y., & Bengio, Y. (1995). Convolutional networks for images, speech, and time series. In M. A. Arbib (Ed.), The handbook of brain theory and neural networks (pp. 255-258). Cambridge, MA: MIT Press.
[21] LeCun Y., Cortes, C., & Burges, C. (1998). The MNIST database. http://yann.lecun.com/exdb/mnist/.
[22] Lennie, P. (2003). The cost of cortical computation. Current Biology, 13, 493-497. ,
[23] Levy, W. B., & Baxter, R. A. (1996). Energy efficient neural codes. Neural Computation, 8, 531-543. ,
[24] Lloyd, S. (1982). Least squares quantization in PCM. IEEE Transactions on Information Theory, 28, 129-137. , · Zbl 0504.94015
[25] McCulloch, W., & Pitts, W. (1943). A logical calculus of the ideas immanent innervous activity. Bulletin of Mathematical Biology, 5, 115-133. · Zbl 0063.03860
[26] Nadal, J. P., & Toulouse, G. (1990). Information storage in sparsely coded memory nets. Network: Computation in Neural Systems, 1, 61-74. , · Zbl 0825.68368
[27] Olshausen, B. A., & Field, D. J. (2004). Sparse coding of sensory inputs. Current Opinion in Neurobiology, 14, 481-487. ,
[28] Palm, G. (1980). On associative memory. Biological Cybernetics, 36, 19-31. , · Zbl 0426.68102
[29] Palm, G. (1981). On the storage capacity of an associative memory with randomly distributed storage elements. Biological Cybernetics, 39, 125-127. , · Zbl 0441.68100
[30] Palm, G. (1982). Neural assemblies: An alternative approach to artificial intelligence. New York: Springer. ,
[31] Palm, G. (1985). Associative networks and their information storage capacity. Cognitive Systems, 1, 107-118.
[32] Palm, G. (1992). On the information storage capacity of local learning rules. Neural Computation, 4, 703-711. ,
[33] Palm, G., Schwenker, F., Sommer, F., & Strey, A. (1997). Neural associative memories. In A. Krikelis & C. Weems (Eds.), Associative processing and processors (pp. 307-325). Washington, DC: IEEE Computer Society.
[34] Sa-Couto, L., & Wichert, A. (2019). Attention inspired network: Steep learning curve in an invariant pattern recognition model. Neural Networks, 114, 38-46. ,
[35] Steinbuch, K. (1961). Die lernmatrix. Biological Cybernetics, 1(1), 36-45. · Zbl 0098.33203
[36] Willshaw, D. J., Buneman, O. P., & Longuet-Higgins, H. C. (1969). Non-holographic associative memory. Nature, 222(5197), 960-962. ,
[37] Wu, X., Kumar, V., Quinlan, J. R., Ghosh, J., Yang, Q., Motoda, H., McLachlan, G., … Steinberg, D. (2008). Top 10 algorithms in data mining. Knowledge and Information Systems, 14(1), 1-37. ,
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.