Analysis and test of efficient methods for building recursive deterministic perceptron neural networks. (English) Zbl 1254.68200

Elizondo, David A.; Birkenhead, Ralph; Góngora, Mario; Taillard, Eric; Luyima, Patrick

Neural Netw. 20, No. 10, 1095-1108 (2007).

Summary: The recursive deterministic perceptron (RDP) feed-forward multilayer neural network is a generalisation of the single layer perceptron topology. This model is capable of solving any two-class classification problem as opposed to the single layer perceptron which can only solve classification problems dealing with linearly separable sets. For all classification problems, the construction of an RDP is done automatically and convergence is always guaranteed. Three methods for constructing RDP neural networks exist: Batch, Incremental, and Modular. The Batch method has been extensively tested and it has been shown to produce results comparable with those obtained with other neural network methods such as back propagation, cascade correlation, rulex, and ruleneg. However, no testing has been done before on the incremental and modular methods. Contrary to the Batch method, the complexity of these two methods is not NP-complete. For the first time, a study on the three methods is presented. This study will allow the highlighting of the main advantages and disadvantages of each of these methods by comparing the results obtained while building RDP neural networks with the three methods in terms of the convergence time, the level of generalisation, and the topology size. The networks were trained and tested using the following standard benchmark classification datasets: IRIS, SOYBEAN, and Wisconsin Breast Cancer. The results obtained show the effectiveness of the Incremental and the Modular methods which are as good as that of the NP-complete batch method but with a much lower complexity level. The results obtained with the RDP are comparable to those obtained with the backpropagation and the cascade correlation algorithms.

MSC:

68T05	Learning and adaptive systems in artificial intelligence
92B20	Neural networks for/in biological studies, artificial life and related topics
62H30	Classification and discrimination; cluster analysis (statistical aspects)

Keywords:

recursive deterministic perceptron; batch learning; incremental learning; modular learning; performance sensitivity analysis; convergence time; generalisation; network topology

Software:

UCI-ml

References:

[1]	Atiya, A., Learning with kernels: support vector machines, regularization, optimization, and beyond, IEEE transactions on neural networks, 16, 3, 781, (2005)
[2]	Bennett, K.P.; Mangasarian, O.L., Robust linear programming discrimination of two linearly inseparable sets, Optimization methods and software, 1, 23-34, (1992)
[3]	Blake, C.L.; Newman, D.J.; Hettich, S.; Merz, C.J., UCI repository of machine learning databases, (1998)
[4]	Boser, B., Guyon, I., & Vapnik, V. (1992). A training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on computational learning theory
[5]	Camastra, Francesco; Verri, Alessandro, A novel kernel method for clustering, IEEE transactions on pattern analysis and machine intelligence, 27, 5, 801-805, (2005)
[6]	Cortes, C.; Vapnik, V., Support-vector network, Machine learning, 20, 273-297, (1995) · Zbl 0831.68098
[7]	Cristianini, N.; Shawe-Taylor, J., An introduction to support vector machines, Vol. I, (2003), Cambridge University Press
[8]	Dasarathy, B.W., Nosing around the neighborhood: A new system structure and classification rule for recognition in partially exposed environments, IEEE transactions on pattern analysis and machine intelligence, 2, 1, 67-71, (1980)
[9]	Dréo, J.; Pétrowski, A.; Siarry, P.; Taillard, E., Metaheuristics for hard optimization methods and case studies, (2006), Springer · Zbl 1118.90058
[10]	Elizondo, D. (1997). The recursive determinist perceptron (RDP) and topology reduction strategies for neural networks. Ph.D. thesis. Université Louis Pasteur, Strasbourg, France
[11]	Elizondo, D. (2004a). Searching for linearly separable subsets using the class of linear separability method. In IEEE-IJCNN (pp. 955-960)
[12]	Elizondo, D. (2004b). Searching for linearly separable subsets using the class of linear separability method. In Proceedings of the IEEE-IJCNN (pp. 955-960)
[13]	Elizondo, D., The linear separability problem: some testing methods, IEEE transactions on neural networks, 17, 2, 330-344, (2006)
[14]	Elizondo, David, Birkenhead, Ralph, & Taillard, Eric (2006). Generalisation and the recursive deterministic perceptron. In IEEE International joined conference on neural networks. Vancouver, Canada · Zbl 1254.68200
[15]	Elizondo, D.A.; Gongora, M.A., Current trends on knowledge extraction and neural networks, ()
[16]	Fisher, R.A., The use of multiple measurements in taxonomic problems, Annual eugenics, 7, II, 179-188, (1936)
[17]	Gates, G.W., The reduced nearest neighbor rule, IEEE transactions on information theory, May, 431-433, (1972)
[18]	Johnson, D.S.; Preparata, F.P., The densest hemisphere problem, Theoretical computer science, 6, 93-107, (1978) · Zbl 0368.68053
[19]	Mangasarian, O.L.; Wolberg, W.H., Cancer diagnosis via linear programming, SIAM news, 23, 5, 1-18, (1990) · Zbl 0726.90096
[20]	Minsky, M.; Papert, S., Perceptrons, (1969), MIT Press Cambridge, MA · Zbl 0197.43702
[21]	Rosenblatt, F., Principles of neurodynamics, (1962), Spartan Washington DC · Zbl 0143.43504
[22]	Sedgewick, R., Algorithms, (1983), Addison-Wesley Publishing Company, (p. 508) · Zbl 0529.68002
[23]	Tajine, M.; Elizondo, D., The recursive deterministic perceptron neural network, Neural networks, 11, 1571-1588, (1997)
[24]	Tajine, M.; Elizondo, D., Growing methods for constructing recursive deterministic perceptron neural networks and knowledge extraction, Artificial intelligence, 102, 295-322, (1998) · Zbl 0909.68138
[25]	Tajine, M.; Elizondo, D.; Fiesler, E.; Korczak, J., The international conference on neural networks, (1997), IEEE
[26]	Weiss, S.M.; Kulikowski, C.A., Computer systems that learn, (1991), Morgan Kaufmann Publishers San Mateo, California
[27]	Wolberg, W.H.; Mangasarian, O.L., Multisurface method of pattern separation for medical diagnosis applied to breast cytology, Proceedings of the national Academy of sciences, 87, December, 9193-9196, (1990) · Zbl 0709.92537