Learning to imitate stochastic time series in a compositional way by chaos.

*(English)*Zbl 1396.68099Summary: This study shows that a mixture of RNN experts model can acquire the ability to generate sequences that are combination of multiple primitive patterns by means of self-organizing chaos. By training the model, each expert learns a primitive sequence pattern, and a gating network learns to imitate stochastic switching of the multiple primitives via chaotic dynamics, utilizing a sensitive dependence on initial conditions. As a demonstration, we present a numerical simulation in which the model learns Markov chain switching among some Lissajous curves by chaotic dynamics. Our analysis shows that by using a sufficient amount of training data, balanced with the network memory capacity, it is possible to satisfy the conditions for embedding the target stochastic sequences into a chaotic dynamical system. It is also shown that reconstruction of a stochastic time series by a chaotic model can be stabilized by adding a negligible amount of noise to the dynamics of the model.

##### MSC:

68T05 | Learning and adaptive systems in artificial intelligence |

37D45 | Strange attractors, chaotic dynamics of systems with hyperbolic behavior |

37M10 | Time series analysis of dynamical systems |

62M10 | Time series, auto-correlation, regression, etc. in statistics (GARCH) |

##### Software:

LSTM
PDF
BibTeX
XML
Cite

\textit{J. Namikawa} and \textit{J. Tani}, Neural Netw. 23, No. 5, 625--638 (2010; Zbl 1396.68099)

Full Text:
DOI

**OpenURL**

##### References:

[1] | Andronov, A.A.; Pontryagin, L.S., Systéms grossiers, Doklady akademii nauk SSSR, 14, 247-251, (1937) · Zbl 0016.11301 |

[2] | Bengio, Y.; Simard, P.; Frasconi, P., Learning long-term dependencies with gradient descent is difficult, IEEE transactions on neural networks, 5, 2, 157-166, (1994) |

[3] | Elman, J.L., Finding structure in time, Cognitive science, 14, 179-211, (1990) |

[4] | Elman, J.L., Distributed representations, simple recurrent networks, and grammatical structure, Machine learning, 7, 195-224, (1991) |

[5] | Evans, G., Semantic theory and tacit knowledge, (), 118-137 |

[6] | Hao, B.L., Elementary symbolic dynamic and chaos in dissipative system, (1989), World Scientific Singapore |

[7] | Hochreiter, S.; Schmidhuber, J., Long short-term memory, Neural computation, 9, 8, 1735-1780, (1997) |

[8] | Ikeda, K.; Otsuka, K.; Matsumoto, K., Maxwell Bloch turbulence, Progress of theoretical physics (supplement), 99, 295-324, (1989) |

[9] | Jacobs, R.A.; Jordan, M.I.; Nowlan, S.J.; Hinton, G.E., Adaptive mixtures of local experts, Neural computation, 3, 1, 79-87, (1991) |

[10] | Jaeger, H., Short term memory in echo state networks, GMD report, 152, 1-60, (2001) |

[11] | Jaeger, H.; Haas, H., Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication, Science, 78-80, (2004) |

[12] | Jordan, M.I., Attractor dynamics and parallelism in a connectionist sequential machine, (), 531-546 |

[13] | Jordan, M.I., Indeterminate motor skill learning problems, () |

[14] | Jordan, M.I.; Jacobs, R.A., Hierarchical mixtures of experts and the EM algorithm, Neural computation, 6, 2, 181-214, (1994) |

[15] | Kaneko, K.; Tsuda, I., Chaotic itinerancy, Chaos: focus issue on chaotic itinerancy, 13, 926-936, (2003) · Zbl 1080.37531 |

[16] | Lind, D.; Marcus, B., An introduction to symbolic dynamics and coding, (1995), Cambridge University Press Cambridge · Zbl 1106.37301 |

[17] | Maass, W., Natschläger, T., & Markram, H. (2002). A fresh look at real-time computation in generic recurrent neural circuits. Technical report. Institute for Theoretical Computer Science, TU Graz |

[18] | Namikawa, J.; Tani, J., A model for learning to segment temporal sequences, utilizing a mixture of RNN experts together with adaptive variance, Neural networks, 21, 1466-1475, (2008) · Zbl 1254.68215 |

[19] | Nishimoto, R.; Tani, J., Learning to generate combinatorial action sequences utilizing the initial sensitivity of deterministic dynamical systems, Neural networks, 17, 925-933, (2004) · Zbl 1068.68123 |

[20] | Pollack, J., The induction of dynamical recognizers, Machine learning, 7, 227-252, (1991) |

[21] | Robinson, C., () |

[22] | Rumelhart, D.E.; Hinton, G.E.; Williams, R.J., Learning internal representations by error propagation, (), 318-362 |

[23] | Schmidhuber, J.; Gers, F.; Eck, D., Learning nonregular languages: A comparison of simple recurrent networks and LSTM, Neural computation, 14, 2039-2041, (2002) · Zbl 1010.68857 |

[24] | Skarada, C.A.; Freeman, W.J., Does the brain make chaos in order to make sense of the world?, Behavioral and brain sciences, 10, 161-165, (1987) |

[25] | Tani, J.; Fukumura, N., Embedding a grammatical description in deterministic chaos: an experiment in recurrent neural learning, Biological cybernetics, 72, 365-370, (1995) |

[26] | Tani, J.; Nolfi, S., Learning to perceive the world as articulated: an approach for hierarchical learning in sensory-motor systems, Neural networks, 12, 1131-1141, (1999) |

[27] | Tsuda, I., Chaotic itinerancy as a dynamical basis of hermeneutics in brain and mind, World futures: the journal of general evolution, 32, 167-184, (1991) |

[28] | Tsuda, I.; Fujii, H., Chaos reality in the brain, Journal of integrative neuroscience, 6, 2, 309-326, (2007) |

[29] | Williams, R.J.; Zipser, D., A learning algorithm for continually running fully recurrent neural networks, Neural computation, 1, 270-280, (1989) |

[30] | Wolpert, D.; Kawato, M., Multiple paired forward and inverse models for motor control, Neural networks, 11, 1317-1329, (1998) |

This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.