Practical selection of SVM parameters and noise estimation for SVM regression. (English) Zbl 1075.68632

Cherkassky, Vladimir; Ma, Yunqian

Neural Netw. 17, No. 1, 113-126 (2004).

Summary: We investigate practical selection of hyper-parameters for support vector machines (SVM) regression (that is, \(\epsilon\)-insensitive zone and regularization parameter \(C\)). The proposed methodology advocates analytic parameter selection directly from the training data, rather than re-sampling approaches commonly used in SVM applications. In particular, we describe a new analytical prescription for setting the value of insensitive zone \(\epsilon\), as a function of training sample size. Good generalization performance of the proposed parameter selection is demonstrated empirically using several low- and high-dimensional regression problems. Further, we point out the importance of Vapnik’s \(\epsilon\)-insensitive loss for regression problems with finite samples. To this end, we compare generalization performance of SVM regression (using proposed selection of \(\epsilon\)-values) with regression using ‘least-modulus’ loss (\(\epsilon=0\)) and standard squared loss. These comparisons indicate superior generalization performance of SVM regression under sparse sample settings, for various types of additive noise.

MSC:

68T05

Learning and adaptive systems in artificial intelligence

Keywords:

Complexity control; Loss function; Parameter selection; Prediction accuracy; Support vector machine regression; VC theory

Software:

ElemStatLearn

References:

[1]	Chapelle, O; Vapnik, V, Model selection for support vector machines, Advances in neural information processing systems, Vol. 12, (1999)
[2]	Cherkassky, V; Ma, Y, Comparison of model selection for regression, Neural computation, 15, 1691-1714, (2003) · Zbl 1046.62001
[3]	Cherkassky, V; Mulier, F, Learning from data: concepts, theory, and methods, (1998), Wiley New York · Zbl 0960.62002
[4]	Cherkassky, V; Shao, X; Mulier, F; Vapnik, V, Model complexity control for regression using VC generalization bounds, IEEE transaction on neural networks, 10, 5, 1075-1089, (1999)
[5]	Drucker, H; Burges, C; Kaufman, L; Smola, A; Vapnik, V, Support vector regression machines, (), 155-161
[6]	Hastie, T; Tibshirani, R; Friedman, J, The elements of statistical learning: data mining, inference and prediction, (2001), Springer Berlin · Zbl 0973.62007
[7]	Huber, P, Robust estimation of a location parameter, Annals of mathematical statistics, 35, 73-101, (1964) · Zbl 0136.39805
[8]	Kwok, J.T, Linear dependency between ε and the input noise in ε-support vector regression, (), 405-410 · Zbl 1001.68888
[9]	Mattera, D; Haykin, S, Support vector machines for dynamic reconstruction of a chaotic system, ()
[10]	Muller, K; Smola, A; Ratsch, G; Schölkopf, B; Kohlmorgen, J; Vapnik, V, Using support vector machines for time series prediction, ()
[11]	Schölkopf, B; Bartlett, P; Smola, A; Williamson, R, Support vector regression with automatic accuracy control, (), 111-116
[12]	Schölkopf, B; Burges, J; Smola, A, Advances in kernel methods: support vector machine, (1999), MIT Press Cambridge, MA
[13]	Schölkopf, B; Smola, A, Learning with kernels: support vector machines, regularization, and beyond, (2002), MIT Press Cambridge, MA
[14]	Smola, A; Murata, N; Schölkopf, B; Muller, K, Asymptotically optimal choice of ε-loss for support vector machines, Proceedings of ICANN 1998, (1998)
[15]	Smola, A., & Schölkopf, B (1998). A tutorial on support vector regression. NeuroCOLT Technical Report NC-TR-98-030, Royal Holloway College, University of London, UK.
[16]	Vapnik, V, Statistical learning theory, (1998), Wiley New York
[17]	Vapnik, V, The nature of statistical learning theory, (1999), Springer Berlin · Zbl 0928.68093
[18]	Vapnik, V (2001). Personal communication.