×

The impact of sparse coding on memory lifetimes in simple and complex models of synaptic plasticity. (English) Zbl 1495.92009

Summary: Models of associative memory with discrete state synapses learn new memories by forgetting old ones. In the simplest models, memories are forgotten exponentially quickly. Sparse population coding ameliorates this problem, as do complex models of synaptic plasticity that posit internal synaptic states, giving rise to synaptic metaplasticity. We examine memory lifetimes in both simple and complex models of synaptic plasticity with sparse coding. We consider our own integrative, filter-based model of synaptic plasticity, and examine the cascade and serial synapse models for comparison. We explore memory lifetimes at both the single-neuron and the population level, allowing for spontaneous activity. Memory lifetimes are defined using either a signal-to-noise ratio (SNR) approach or a first passage time (FPT) method, although we use the latter only for simple models at the single-neuron level. All studied models exhibit a decrease in the optimal single-neuron SNR memory lifetime, optimised with respect to sparseness, as the probability of synaptic updates decreases or, equivalently, as synaptic complexity increases. This holds regardless of spontaneous activity levels. In contrast, at the population level, even a low but nonzero level of spontaneous activity is critical in facilitating an increase in optimal SNR memory lifetimes with increasing synaptic complexity, but only in filter and serial models. However, SNR memory lifetimes are valid only in an asymptotic regime in which a mean field approximation is valid. By considering FPT memory lifetimes, we find that this asymptotic regime is not satisfied for very sparse coding, violating the conditions for the optimisation of single-perceptron SNR memory lifetimes with respect to sparseness. Similar violations are also expected for complex models of synaptic plasticity.

MSC:

92B20 Neural networks for/in biological studies, artificial life and related topics
91E40 Memory and learning in psychology
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Andersen P, Morris RGM, Amaral D, Bliss TVP, O’Keefe J (2007) The hippocampus book. Oxford University Press, Oxford
[2] Amit, D.; Fusi, S., Learning in neural networks with material synapses, Neural Comput, 6, 957-982 (1994) · doi:10.1162/neco.1994.6.5.957
[3] Andrews, G.; Askey, R.; Roy, R., Special functions (1999), Cambridge: Cambridge University Press, Cambridge · Zbl 0920.33001 · doi:10.1017/CBO9781107325937
[4] Appleby, P.; Elliott, T., Stable competitive dynamics emerge from multispike interactions in a stochastic model of spike-timing-dependent plasticity, Neural Comput, 18, 2414-2464 (2006) · Zbl 1107.92010 · doi:10.1162/neco.2006.18.10.2414
[5] Bagal, A.; Kao, J.; Tang, CM; Thompson, S., Long-term potentiation of exogenous glutamate responses at single dendritic spines, Proc Natl Acad Sci USA, 102, 14434-14439 (2005) · doi:10.1073/pnas.0501956102
[6] Barrett, A.; van Rossum, M., Optimal learning rules for discrete synapses, PLoS Comput Biol, 4, e1000230 (2008) · doi:10.1371/journal.pcbi.1000230
[7] Bartol, T.; Bromer, C.; Kinney, J.; Chirillo, M.; Bourne, J.; Harris, K.; Sejnowski, T., Nanoconnectomic upper bound on the variability of synaptic plasticity, Elife, 4, e10778 (2015) · doi:10.7554/eLife.10778
[8] Bienenstock, E.; Cooper, L.; Munro, P., Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex, J Neurosci, 2, 32-48 (1982) · doi:10.1523/JNEUROSCI.02-01-00032.1982
[9] Bliss, T.; Lømo, T., Long-lasting potentiation of synaptic transmission in the dentate area of the unanaesthetized rabbit following stimulation of the perforant path, J Physiol, 232, 331-356 (1973) · doi:10.1113/jphysiol.1973.sp010273
[10] Burkitt, A.; Meffin, H.; Grayden, D., Spike-timing-dependent plasticity: the relationship to rate-based learning for models with weight dynamics determined by a stable fixed point, Neural Comput, 16, 885-940 (2004) · Zbl 1054.92007 · doi:10.1162/089976604773135041
[11] Csicsvari, J.; Hirase, H.; Mamiya, A.; Buzsaki, G., Ensemble patterns of hippocampal CA3-CA1 neurons during sharp wave-associated population events, Neuron, 28, 585-594 (2000) · doi:10.1016/S0896-6273(00)00135-5
[12] Eichenbaum, H.; Cohen, NJ, From conditioning to conscious recollection (2001), Oxford: Oxford University Press, Oxford
[13] Elliott, T., Temporal dynamics of rate-based plasticity rules in a stochastic model of spike-timing-dependent plasticity, Neural Comput, 20, 2253-2307 (2008) · Zbl 1140.92003 · doi:10.1162/neco.2008.06-07-555
[14] Elliott, T., Memory nearly on a spring: a mean first passage time approach to memory lifetimes, Neural Comput, 26, 1873-1923 (2014) · Zbl 1417.91421 · doi:10.1162/NECO_a_00622
[15] Elliott T (2016a) The enhanced rise and delayed fall of memory in a model of synaptic integration: extension to discrete state synapses. Neural Comput 28:1927-1984 · Zbl 1414.91318
[16] Elliott T (2016b) Variations on the theme of synaptic filtering: a comparison of integrate-and-express models of synaptic plasticity for memory lifetimes. Neural Comput 28:2393-2460 · Zbl 1414.92089
[17] Elliott T (2017a) First passage time memory lifetimes for simple, multistate synapses. Neural Comput 29:3219-3259 · Zbl 1414.91319
[18] Elliott T (2017b) Mean first passage memory lifetimes by reducing complex synapses to simple synapses. Neural Comput 29:1468-1527 · Zbl 1414.92090
[19] Elliott, T., First passage time memory lifetimes for simple, multistate synapses: beyond the eigenvector requirement, Neural Comput, 31, 8-67 (2019) · Zbl 1471.92068 · doi:10.1162/neco_a_01147
[20] Elliott, T., First passage time memory lifetimes for multistate, filter-based synapses, Neural Comput, 32, 1069-1143 (2020) · Zbl 1475.92012 · doi:10.1162/neco_a_01283
[21] Elliott, T.; Lagogiannis, K., Taming fluctuations in a stochastic model of spike-timing-dependent plasticity, Neural Comput, 21, 3363-3407 (2009) · Zbl 1180.92013 · doi:10.1162/neco.2009.12-08-916
[22] Elliott, T.; Lagogiannis, K., The rise and fall of memory in a model of synaptic integration, Neural Comput, 24, 2604-2654 (2012) · Zbl 1268.92020 · doi:10.1162/NECO_a_00335
[23] Fusi, S.; Drew, P.; Abbott, L., Cascade models of synaptically stored memories, Neuron, 45, 599-611 (2005) · doi:10.1016/j.neuron.2005.02.001
[24] Hopfield, J., Neural networks and physical systems with emergent collective computational abilities, Proc Natl Acad Sci USA, 79, 2554-2558 (1982) · Zbl 1369.92007 · doi:10.1073/pnas.79.8.2554
[25] Huang, Y.; Amit, Y., Precise capacity analysis in binary networks with multiple coding level inputs, Neural Comput, 22, 660-688 (2010) · Zbl 1185.92013 · doi:10.1162/neco.2009.02-09-967
[26] Lahiri, S.; Ganguli, S.; Burges, C.; Bottou, L.; Welling, M.; Ghahramani, Z.; Weinberger, K., A memory frontier for complex synapses, Advances in neural information processing systems, 1034-1042 (2013), Cambridge: MIT Press, Cambridge
[27] Leibold, C.; Kempter, R., Memory capacity for sequences in a recurrent network with biological constraints, Neural Comput, 18, 904-941 (2006) · Zbl 1087.92007 · doi:10.1162/neco.2006.18.4.904
[28] Leibold, C.; Kempter, R., Sparseness constrains the prolongation of memory lifetime via synaptic metaplasticity, Cereb Cortex, 18, 67-77 (2008) · doi:10.1093/cercor/bhm037
[29] Lynch, G.; Dunwiddie, T.; Gribkoff, V., Heterosynaptic depression: a postsynaptic correlate of long-term potentiation, Nature, 266, 737-739 (1977) · doi:10.1038/266737a0
[30] Montgomery, J.; Madison, D., State-dependent heterogeneity in synaptic depression between pyramidal cell pairs, Neuron, 33, 765-777 (2002) · doi:10.1016/S0896-6273(02)00606-2
[31] Montgomery, J.; Madison, D., Discrete synaptic states define a major mechanism of synapse plasticity, Trends Neurosci, 27, 744-750 (2004) · doi:10.1016/j.tins.2004.10.006
[32] Nadal, J.; Toulouse, G.; Changeux, J.; Dehaene, S., Networks of formal neurons and memory palimpsests, Europhys Lett, 1, 535-542 (1986) · doi:10.1209/0295-5075/1/10/008
[33] O’Connor D, Wittenberg G, Wang SH (2005) Dissection of bidirectional synaptic plasticity into saturable unidirectional process. J Neurophysiol 94:1565-1573
[34] O’Connor D, Wittenberg G, Wang SH (2005) Graded bidirectional synaptic plasticity is composed of switch-like unitary events. Proc Natl Acad Sci USA 102:9679-9684
[35] Olshausen, BA; Field, DJ, Sparse coding of sensory inputs, Curr Opin Neurobiol, 14, 481-487 (2004) · doi:10.1016/j.conb.2004.07.007
[36] Parisi, G., A memory which forgets, J Phys A: Math Gen, 19, L617-L620 (1986) · doi:10.1088/0305-4470/19/10/011
[37] Petersen, C.; Malenka, R.; Nicoll, R.; Hopfield, J., All-or-none potentiation at CA3-CA1 synapses, Proc Natl Acad Sci USA, 95, 4732-4737 (1998) · doi:10.1073/pnas.95.8.4732
[38] Rao-Ruiz, P.; Visser, E.; Mitric, M.; Smit, AB; van den Oever, MC, A synaptic framework for the persistence of memory engrams, Front Synaptic Neurosci, 13, 661476 (2021) · doi:10.3389/fnsyn.2021.661476
[39] Richards, BA; Frankland, PW, The persistence and transience of memory, Neuron, 94, 1071-1084 (2017) · doi:10.1016/j.neuron.2017.04.037
[40] Rubin, D.; Fusi, S., Long memory lifetimes require complex synapses and limited sparseness, Front Comput Neurosci, 1, 7 (2007) · doi:10.3389/neuro.01.1.1.001.2007
[41] Sobczyk, A.; Svoboda, K., Activity-dependent plasticity of the NMDA-receptor fractional \(Ca^{2+}\) current, Neuron, 53, 17-24 (2007) · doi:10.1016/j.neuron.2006.11.016
[42] Tsodyks, M., Associative memory in neural networks with binary synapses, Mod Phys Lett B, 4, 713-716 (1990) · doi:10.1142/S0217984990000891
[43] Tsodyks M, Feigel’man M (1988) The enhanced storage capacity in neural networks with low activity levels. Europhys Lett 6:101-105
[44] Uhlenbeck, G.; Ornstein, L., On the theory of Brownian motion, Phys Rev, 36, 823-841 (1930) · JFM 56.1277.03 · doi:10.1103/PhysRev.36.823
[45] van Kampen, N., Stochastic processes in physics and chemistry (1992), Amsterdam: Elsevier, Amsterdam · Zbl 0511.60038
[46] Yasuda, R.; Sabatini, B.; Svoboda, K., Plasticity of calcium channels in dendritic spines, Nat Neurosci, 6, 948-955 (2003) · doi:10.1038/nn1112
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.