Learning algorithms for neural networks and neuro-fuzzy systems with separable structures.

*(English. Russian original)*Zbl 1317.93162
Cybern. Syst. Anal. 51, No. 2, 173-186 (2015); translation from Kibern. Sist. Anal. 2015, No. 2, 13-28 (2015).

Summary: This article considers the problem of training neural networks and neuro-fuzzy systems, which lead to separable models, i.e., structures nonlinear with respect to some unknown parameters and linear with respect to others. New algorithms for training them are proposed that are based on a nonlinear optimization problem including a priori information only on nonlinear input parameters. It is assumed that this information can be obtained from a training set, the distribution of a generating set, or linguistic information. To solve the problem, the Gauss-Newton method with linearization in the vicinity of the last estimate, asymptotic representations of the pseudo-inverse of perturbed matrices, and separable structures of models are used. The obtained algorithms have the following important properties: they do not require the selection of initial values of linearly entering parameters, which can lead to divergence, but, at the same time, it is not necessary to find partial derivatives of a
projection matrix; they can be used in serial and batch processing; well-known algorithms are obtained from them as special cases, and a simulation has shown that the proposed algorithms can outperform the former in accuracy and convergence rate.

##### MSC:

93C42 | Fuzzy control/observation systems |

92B20 | Neural networks for/in biological studies, artificial life and related topics |

68T05 | Learning and adaptive systems in artificial intelligence |

PDF
BibTeX
XML
Cite

\textit{B. A. Skorohod}, Cybern. Syst. Anal. 51, No. 2, 173--186 (2015; Zbl 1317.93162); translation from Kibern. Sist. Anal. 2015, No. 2, 13--28 (2015)

Full Text:
DOI

##### References:

[1] | S. Haykin, Neural Networks and Learning Machines, Prentice Hall, Upper Saddle River, NJ (2009). |

[2] | Golub, GH; Pereyra, V, The differentiation of pseudoinverses and nonlinear least squares problems whose variables separate, SIAM J. Numer. Anal., 10, 413-432, (1973) · Zbl 0258.65045 |

[3] | Pereyra, V; Scherer, G; Wong, F, Variable projections neural network training, Math. and Comput. Simulation, 73, 231-243, (2006) · Zbl 1104.65060 |

[4] | J. Sjoberg and M. Viberg, “Separable non-linear least-squares minimization and possible improvements for neural net fitting,” in: IEEE Workshop in Neural Networks for Signal Processing, Florida (USA) (1997), pp. 345-354. |

[5] | L. Ngia, “Separable Nonlinear least-squares methods for on-line estimation of neural nets Hammerstein models,” in: Proc. IEEE Signal Processing Society Workshop, 1, Sydney (2000), pp. 65-74. |

[6] | P. Parisi, D. Di. Claudio, G. Orlandi, and B. D. Rao, “A generalized learning paradigm exploiting the structure of feedforward neural networks,” in: IEEE Trans. Neural Networks, 7 (1996), pp. 1450-1460. · Zbl 1298.68228 |

[7] | Jang, JSR, ANFIS: adaptive-network-based fuzzy inference system, IEEE Trans. on Systems, Man, and Cybernetics, 3, 665-685, (1993) |

[8] | Bodyanskiy, Y; Kolodyazhniy, VV; Stephan, A, An adaptive learning algorithm for a neuro-fuzzy network, 68-75, (2001), Berlin-Heidelberg-New York · Zbl 1043.68650 |

[9] | Bodyanskiy, Y; Kolodyazhniy, VV; Otto, P, Neuro-fuzzy kolmogorov’s network for time-series prediction and pattern classification, No. 3698, 191-202, (2005), Heidelberg · Zbl 1137.68524 |

[10] | Bodyanskiy, YV; Pliss, I; Vynokurova, O, Adaptive wavelet-neuro-fuzzy network in the forecasting and emulation tasks, Intern. J. Inform. Theories & Appl., 15, 47-55, (2008) |

[11] | Huang, GB; Wang, DH; Lan, Y, Extreme learning machines: A survey, Intern. J. of Machine Leaning and Cybernetics, 2, 107-122, (2011) |

[12] | Kim, C-T; Lee, J-J, Training two-layered feedforward networks with variable projection method, IEEE Trans. Neural Networks, 19, 371-375, (2008) |

[13] | Skorohod, BA, Diffusion learning algorithms for feedforward neural networks, Cybernetics and Systems Analysis, 49, 334-347, (2013) · Zbl 1298.68228 |

[14] | V. Kecman, Learning and Soft Computing, Support Vector Machines, Neural Networks, and Fuzzy Logic Models, MIT Press, Cambridge (Mass.) (2001). · Zbl 0994.68109 |

[15] | B. A. Skorohod, “Asymptotics of linear recurrent regression under diffuse initialization,” Problems of Control and Informatics, No. 3, 98-107 (2009). |

[16] | R. Jang, C. Sun, and E. Mizutani, Neuro-Fuzzy and Soft Computing: A Computational Approach to Learning and Machine Intelligence, Prentice Hall, Upper Saddle River (1997). |

[17] | A. Albert, Regression, Pseudoinversion, and Recurrent Estimation [Russian translation], Nauka, Moscow (1977). |

[18] | H. R. Wimmer, “Stabilizing and unmixed solutions of the discrete time algebraic Riccati equation,” in: Proc. Workshop on the Riccati Equation in Control, Systems, and Signals, Italy (1989), pp. 95-98. |

[19] | Bertsekas, DP, Incremental least squares methods and the extended Kalman filter, SIAM J. Optimization, 3, 807-822, (1996) · Zbl 0945.93026 |

[20] | Curve Fitting Toolbox 3. The MathWorks, Inc. |

[21] | Fuzzy Logic Toolbox. The MathWorks, Inc. |

[22] | Narendra, KS; Parthasarathy, K, Identification and control of dynamical systems using neural networks, IEEE Trans. Neural Networks, 1, 4-27, (1990) |

This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.