Resonator networks. I: An efficient solution for factoring high-dimensional, distributed representations of data structures.

*(English)*Zbl 07290009Summary: The ability to encode and manipulate data structures with distributed neural representations could qualitatively enhance the capabilities of traditional neural networks by supporting rule-based symbolic reasoning, a central property of cognition. Here we show how this may be accomplished within the framework of Vector Symbolic Architectures (VSAs) (Plate, 1991; Gayler, 1998; Kanerva, 1996), whereby data structures are encoded by combining high-dimensional vectors with operations that together form an algebra on the space of distributed representations. In particular, we propose an efficient solution to a hard combinatorial search problem that arises when decoding elements of a VSA data structure: the factorization of products of multiple codevectors. Our proposed algorithm, called a resonator network, is a new type of recurrent neural network that interleaves VSA multiplication operations and pattern completion. We show in two examples – parsing of a tree-like data structure and parsing of a visual scene – how the factorization problem arises and how the resonator network can solve it. More broadly, resonator networks open the possibility of applying VSAs to myriad artificial intelligence problems in real-world domains. The companion article in this issue (Kent, Frady, Sommer, & Olshausen, 2020) presents a rigorous analysis and evaluation of the performance of resonator networks, showing it outperforms alternative approaches.

Reviewer: Reviewer (Berlin)

##### MSC:

68 | Computer science |

PDF
BibTeX
XML
Cite

\textit{E. P. Frady} et al., Neural Comput. 32, No. 12, 2311--2331 (2020; Zbl 07290009)

Full Text:
DOI

##### References:

[1] | Adelson, E. H., & Pentland, A. P. (1996). The perception of shading and reflectance. In D. Knill & W. Richards (Eds.), Perception as Bayesian inference (pp. 409-423). New York: Cambridge University Press. , |

[2] | Anderson, A. G., Ratnam, K., Roorda, A., & Olshausen, B. (2020). High acuity vision from retinal image motion. Journal of Vision, 20(7), 34. , |

[3] | Barron, J. T., & Malik, J. (2014). Shape, illumination, and reflectance from shading. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(8), 1670-1687. , |

[4] | Barrow, H., & Tenenbaum, J. (1978). Recovering intrinsic scene characteristics from images. Computer Vision Systems, 2, 3-26. |

[5] | Cadieu, C. F., & Olshausen, B. A. (2012). Learning intermediate-level representations of form and motion from natural movies. Neural Computation, 24(4), 827-866. , |

[6] | Cox, G. E., Kachergis, G., Recchia, G., & Jones, M. N. (2011). Toward a scalable holographic word-form representation. Behavior Research Methods, 43(3), 602-615. , |

[7] | da Silva, A. P., Comon, P., & de Almeida, A. L. (2015). An iterative deflation algorithm for exact CP tensor decomposition. In Proceedings of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 3961-3965). Piscataway, NJ: IEEE. , |

[8] | Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. |

[9] | Feldman, J. (2013). The neural binding problem(s). Cognitive Neurodynamics, 7, 1-11. , |

[10] | Fodor, J. A. (1975). The language of thought. New York: Crowell. |

[11] | Fodor, J. A., & Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2), 3-71. , |

[12] | Frady, E. P., Kleyko, D., & Sommer, F. T. (2018). A theory of sequence indexing and working memory in recurrent neural networks. Neural Computation, 30, 1449-1513. , |

[13] | Frady, E., Kleyko, D., & Sommer, F. (2020). Variable binding for sparse distributed representations: Theory and applications. arXiv:209.06734. |

[14] | Frady, E. P., & Sommer, F. T. (2019). Robust computation with rhythmic spike patterns. In Proceedings of the National Academy of Sciences, 116(36), 18050-18059. , |

[15] | Gayler, R. W. (1998). Multiplicative binding, representation operators and analogy [workshop poster]. In K. Holyoak, D. Gentner, & B. Kokinov (Eds.), Advances in analogy research: Integration of theory and data from the cognitive, computational, and neural sciences. Berlin: Springer. |

[16] | Gayler, R. (2003). Vector symbolic architectures answer Jackendoff’s challenges for cognitive neuroscience. In Proceedings of the ICCS/ASCS International Conference on Cognitive Science. Amsterdam: Elsevier Procedia. |

[17] | Gayler, R. W., & Levy, S. D. (2009). A distributed basis for analogical mapping. In New Frontiers in Analogy Research: Proceedings of the Second International Analogy Conference-Analogy, vol. 9 (pp. 165-174). NBU Press. |

[18] | Hinton, G. E. (1990). Mapping part-whole hierarchies into connectionist networks. Artificial Intelligence, 46(1-2), 47-75. , |

[19] | Hopfield, J. J. (1982). Neural networks and physical systems with emergent collective computational abilities. In Proceedings of the National Academy of Sciences, 79(8), 2554-2558. , · Zbl 1369.92007 |

[20] | Hummel, J. E., & Holyoak, K. J. (1997). Distributed representations of structure: A theory of analogical access and mapping. Psychological Review, 104(3), 427. , |

[21] | Jackendoff, R. (2002). Foundations of language: Brain, meaning, grammar, evolution. Oxford: Oxford University Press. , |

[22] | Joshi, A., Halseth, J. T., & Kanerva, P. (2016). Language geometry using random indexing. In Proceedings of the International Symposium on Quantum Interaction (pp. 265-274). Berlin: Springer. |

[23] | Kanerva, P. (1996). Binary spatter-coding of ordered k-tuples. In Proceedings of the International Conference on Artificial Neural Networks (pp. 869-873). Berlin: Springer. , |

[24] | Kanerva, P. (1997). Fully distributed representation. In Proceedings of the 1997 Real World Computing Symposium (pp. 358-365). Real World Computing Partnership. |

[25] | Kanerva, P. (1998). Large patterns make great symbols: An example of learning from example. In Proceedings of the International Workshop on Hybrid Neural Systems (pp. 194-203). Berlin: Springer. |

[26] | Kanerva, P. (2009). Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors. Cognitive Computation, 1(2), 139-159. , |

[27] | Kent, S. J., Frady, E. P., Sommer, F. T., & Olshausen, B. A. (2020). Resonator networks, 2: Factorization performance and capacity compared to optimization-based methods. Neural Computation, 32(12), 2332-2388. , |

[28] | Kleyko, D., Rahimi, A., Rachkovskij, D. A., Osipov, E., & Rabaey, J. M. (2018). Classification and recall with binary hyperdimensional computing: Tradeoffs in choice of density and mapping characteristics. IEEE Transactions on Neural Networks and Learning Systems, 29(12), 5880-5898. , |

[29] | Laiho, M., Poikonen, J. H., Kanerva, P., & Lehtonen, E. (2015). High-dimensional computing with sparse vectors. In Proceedings of the 2015 IEEE Biomedical Circuits and Systems Conference (pp. 1-4). Piscataway, NJ: IEEE. |

[30] | Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017). Building machines that learn and think like people. Behavioral and Brain Sciences, 40. |

[31] | LeCun, Y. (1998). The MNIST database of handwritten digits. |

[32] | Ledoux, M. (2001). The concentration of measure phenomenon. Providence, RI: American Mathematical Society. · Zbl 0995.60002 |

[33] | McClelland, J. L., Rumelhart, D. E., & PDP Research Group. (1986). Parallel distributed processing: Explorations in the Microstructure of Cognition 1. Cambridge, MA: MIT Press. |

[34] | Mel, B. W., & Koch, C. (1990). Sigma-Pi learning: On radial basis functions and cortical associative learning. In D. S. Touretzky (Ed.), Advances in neural information processing systems, 2 (pp. 474-481). San Mateo, CA: Morgan Kaufmann. |

[35] | Memisevic, R., & Hinton, G. E. (2010). Learning to represent spatial transformations with factored higher-order Boltzmann machines. Neural Computation, 22(6), 1473-1492. , · Zbl 1202.68317 |

[36] | Miller, G. A. (1956). The magic number seven plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 91-97. , |

[37] | Nickel, M., Tresp, V., & Kriegel, H.-P. (2011). A three-way model for collective learning on multi-relational data. In Proceedings of the 28th International Conference on Machine Learning. |

[38] | Plate, T. A. (1991). Holographic reduced representations: Convolution algebra for compositional distributed representations. In Proceedings of the International Joint Conference on Artificial Intelligence (pp. 30-35). San Mateo, CA: Morgan Kaufmann. · Zbl 0744.68044 |

[39] | Plate, T. A. (1995). Holographic reduced representations. IEEE Transactions on Neural Networks, 6(3), 623-641. , |

[40] | Plate, T. A. (2000a). Analogy retrieval and processing with distributed vector representations. Expert Systems, 17(1), 29-40. , |

[41] | Plate, T. A. (2000b). Randomly connected sigma-pi neurons can form associator networks. Network: Computation in Neural Systems, 11(4), 321-332. , · Zbl 1037.92004 |

[42] | Plate, T. A. (2003). Holographic reduced representation: Distributed representation of cognitive structure. Stanford, CA: CSLI Publications. |

[43] | Rachkovskij, D. A., & Kussul, E. M. (2001). Binding and normalization of binary sparse distributed representations by context-dependent thinning. Neural Computation, 13(2), 411-452. , · Zbl 0973.68204 |

[44] | Räsänen, O. J. (2015). Generating hyperdimensional distributed representations from continuous-valued multivariate sensory input. In Proceedings of the 37th Annual Meeting of the Cognitive Science Society.Red Hook, NY: Curran. |

[45] | Smolensky, P. (1990). Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial Intelligence, 46(1-2), 159-216. , · Zbl 0717.68095 |

[46] | Socher, R., Chen, D., Manning, C. D., & Ng, A. (2013). Reasoning with neural tensor networks for knowledge base completion. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, & K. Q. Weinberger (Eds.), Advances in neural information processing systems, 26 (pp. 926-934). Red Hook, NY: Curran. |

[47] | Treisman, A. M., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97-136. , |

[48] | Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., … Polosukhin, I. (2017). Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in neural information processing systems, 30 (pp. 5998-6008). Red Hook, NY: Curran. |

[49] | von der Malsburg, C. (1999). The what and why of binding: The modeler’s perspective. Neuron, 24(1), 95-104. , |

[50] | Wolfe, J. M., & Cave, K. R. (1999). The psychophysical evidence for a binding problem in human vision. Neuron, 24(1), 11-17. , |

This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.