zbMATH — the first resource for mathematics

Geometry of learning: Visualizing the performance of neural network supervised training methods. (English) Zbl 0898.68075
Summary: The empirical comparison of numerical methods is of great importance in the development of improved training methods and in algorithm selection for problem solving since it is not always evident which algorithm is proper for a given class of applications. Questions concerning “the cost”, in terms of function evaluations, the speed of convergence, in terms of epochs and the sensitivity of an algorithm to initial conditions are usually addressed by practitioners of the field. To this end a software package for analyzing and visualizing the convergence behavior of training methods is introduced in this contribution. This package gives quantitative measures for the terms “cost”, “fast” and “sensitive”. Also it gives pieces of information and displays the geometry of basins of attraction for any training method. It displays also, using different colors, the regions of convergence to the minima for various training methods. Moreover, it indicates the rate of their convergence as well as the region of divergence of these methods. Furthermore, this package gives statistical information regarding the total convergence area in a specific domain for various minima.

68T15 Theorem proving (deduction, resolution, etc.) (MSC2010)
problem solving
Full Text: DOI
[1] Haykin, S., ()
[2] Rumelhart, D.E.; Hinton, G.E.; Williams, R.J., Learning internal representations by error propagation, (), 318-362
[3] Magoulas, G.D.; Vrahatis, M.N.; Androulakis, G.S., Effective backpropagation training with variable stepsize, Neural networks, (1996), in press · Zbl 0884.68106
[4] Johansson, E.M.; Dowala, F.U.; Goodman, D.M., Back-propagation learning for multilayer feed forward neural networks using the conjugate gradient method, Int. J. of neural systems, 2, 291-302, (1991)
[5] Van Der Smagt, P.P., Minimisation methods for training feedforward neural networks, Neural networks, 7, 1-11, (1994)
[6] Hagan, M.T.; Menhaj, M.B., Training feedforward networks with the marqnardt algorithm, IEEE trans. neural networks, 5, 989-993, (1994)
[7] Magoulas, G.D.; Vrahatis, M.N.; Grapsa, T.N.; Androulakis, G.S., Neural network supervised training based on a dimension reducing method, Annals of mathematics and artificial intelligence, (1996), accepted for publication · Zbl 0884.68106
[8] Androulakis, G.S.; Vrahatis, M.N., OPTAC: a portable software package for analyzing and comparing optimization methods by visualizaton, Journal of computational and applied mathematics, 72, 41-62, (1996) · Zbl 0857.65066
[9] Jacobs, R.A., Increased rates of convergence through learning rate adaptation, Neural networks, 1, 295-307, (1988)
[10] Press, W.H.; Teukolsky, S.A.; Vetterling, W.T.; Flannery, B.P., ()
[11] Nørgaard, M., Neural network based system identification toolbox, ()
[12] Chen, S.; Cowan, C.F.N.; Billings, S.A.; Grant, P.M., A parallel recursive prediction error algorithm for training layered neural network, Int. J. control, 51, 1215-1228, (1990)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.