Analysis of support vector machines regression. (English) Zbl 1185.68577

Tong, Hongzhi; Chen, Di-Rong; Peng, Lizhong

Found. Comput. Math. 9, No. 2, 243-257 (2009).

Summary: Support vector machines regression (SVMR) is a regularized learning algorithm in reproducing kernel Hilbert spaces with a loss function called the \(\varepsilon \)-insensitive loss function. Compared with the well-understood least square regression, the study of SVMR is not satisfactory, especially the quantitative estimates of the convergence of this algorithm. This paper provides an error analysis for SVMR, and introduces some recently developed methods for analysis of classification algorithms such as the projection operator and the iteration technique. The main result is an explicit learning rate for the SVMR algorithm under some assumptions.

MSC:

68T05	Learning and adaptive systems in artificial intelligence
62J02	General nonlinear regression

References:

[1]	N. Aronszajn, Theory of reproducing kernels, Trans. Am. Math. Soc. 68, 337–404 (1950). · Zbl 0037.20701
[2]	P.L. Bartlett, The sample complexity of pattern classification with neural networks: The size of the weights is more import than the size of the network, IEEE Trans. Inf. Theory 44, 525–536 (1998). · Zbl 0901.68177
[3]	O. Bousquet, A. Elisseeff, Stability and generalization, J. Mach. Learn. Res. 2, 499–526 (2002). · Zbl 1007.68083
[4]	D.R. Chen, Q. Wu, Y. Ying, D.X. Zhou, Support vector machine soft margin classifiers: error analysis, J. Mach. Learn. Res. 5, 1143–1175 (2004). · Zbl 1222.68167
[5]	A. Christmann, I. Steinwart, Consistency and robustness of kernel-based regression in convex risk minimization, Bernoulli 13, 799–819 (2007). · Zbl 1129.62031
[6]	F. Cucker, S. Smale, On the mathematical foundations of learning theory, Bull. Am. Math. Soc. 39, 1–49 (2001). · Zbl 0983.68162
[7]	F. Cucker, S. Smale, Best choices for regularization parameters in learning theory: On the bias-variance problem, Found. Comput. Math. 2, 413–428 (2002). · Zbl 1057.68085
[8]	E. De Vito, A. Caponnetto, L. Rosasco, Model selection for regularized least-squares algorithm in learning theory, Found. Comput. Math. 5, 59–85 (2005). · Zbl 1083.68106
[9]	L. Devroye, L. Györfi, G. Lugosi, A Probabilistic Theory of Pattern Recognition (Springer, New York, 1997).
[10]	T. Evgeniou, M. Pontil, T. Poggio, Regularization networks and support vector machines, Adv. Comput. Math. 13, 1–50 (2000). · Zbl 0939.68098
[11]	P.J. Huber, Robust Statistics (Wiley, New York, 1981). · Zbl 0536.62025
[12]	T. Poggio, S. Smale, The mathematics of learning: Deal with data, Not. Am. Math. Soc. 50, 537–544 (2003). · Zbl 1083.68100
[13]	M. Pontil, S. Mukherjee, F. Girosi, On the noise model of support vector machine regression, A.I. Memo 1651, MIT Artificial Intelligence Lab., 1998.
[14]	L. Rosasco, E. De Vito, A. Caponnetto, M. Piana, A. Verri, Are loss functions all the same? Neural Comput. 16, 1063–1076 (2004). · Zbl 1089.68109
[15]	B. Scholkopf, A.J. Smola, Learning with Kernel (MIT Press, Cambridge, 2002).
[16]	S. Smale, D.X. Zhou, Shannon sampling II. Connections to learning theory, Appl. Comput. Harmon. Anal. 19, 285–302 (2005). · Zbl 1107.94008
[17]	S. Smale, D.X. Zhou, Learning theory estimates via integral operators and their applications, Constr. Approx. 26, 153–172 (2007). · Zbl 1127.68088
[18]	C. Scovel, I. Steinwart, Fast rates for support vector machine, in Proceedings of the Conference on Learning Theory (COLT-2005), pp. 279–294. · Zbl 1137.68564
[19]	V. Vapnik, The Nature of Statistical Learning Theory (Springer, New York, 1995). · Zbl 0833.62008
[20]	V. Vapnik, Statistical Learning Theory (Wiley, New York, 1998). · Zbl 0935.62007
[21]	Q. Wu, D.X. Zhou, SVM soft margin classifiers: Linear programming versus quadratic programming, Neural Comput. 17, 1160–1187 (2005). · Zbl 1108.90324
[22]	Q. Wu, Y. Ying, D.X. Zhou, Learning rates of least-square regularized regression, Found. Comput. Math. 6, 171–192 (2006). · Zbl 1100.68100
[23]	D.X. Zhou, The covering number in learning theory, J. Complex. 18, 739–767 (2002). · Zbl 1016.68044
[24]	D.X. Zhou, Capacity of reproducing kernel spaces in learning theory, IEEE Trans. Inf. Theory 49, 1743–1752 (2003). · Zbl 1290.62033
[25]	D.X. Zhou, K. Jetter, Approximation with polynomial kernels and SVM classifiers, Adv. Comput. Math. 25, 323–344 (2006). · Zbl 1095.68103