Practical selection of SVM parameters and noise estimation for SVM regression. (English) Zbl 1075.68632

Summary: We investigate practical selection of hyper-parameters for support vector machines (SVM) regression (that is, \(\epsilon\)-insensitive zone and regularization parameter \(C\)). The proposed methodology advocates analytic parameter selection directly from the training data, rather than re-sampling approaches commonly used in SVM applications. In particular, we describe a new analytical prescription for setting the value of insensitive zone \(\epsilon\), as a function of training sample size. Good generalization performance of the proposed parameter selection is demonstrated empirically using several low- and high-dimensional regression problems. Further, we point out the importance of Vapnik’s \(\epsilon\)-insensitive loss for regression problems with finite samples. To this end, we compare generalization performance of SVM regression (using proposed selection of \(\epsilon\)-values) with regression using ‘least-modulus’ loss (\(\epsilon=0\)) and standard squared loss. These comparisons indicate superior generalization performance of SVM regression under sparse sample settings, for various types of additive noise.


68T05 Learning and adaptive systems in artificial intelligence


Full Text: DOI


[1] Chapelle, O; Vapnik, V, Model selection for support vector machines, Advances in neural information processing systems, Vol. 12, (1999)
[2] Cherkassky, V; Ma, Y, Comparison of model selection for regression, Neural computation, 15, 1691-1714, (2003) · Zbl 1046.62001
[3] Cherkassky, V; Mulier, F, Learning from data: concepts, theory, and methods, (1998), Wiley New York · Zbl 0960.62002
[4] Cherkassky, V; Shao, X; Mulier, F; Vapnik, V, Model complexity control for regression using VC generalization bounds, IEEE transaction on neural networks, 10, 5, 1075-1089, (1999)
[5] Drucker, H; Burges, C; Kaufman, L; Smola, A; Vapnik, V, Support vector regression machines, (), 155-161
[6] Hastie, T; Tibshirani, R; Friedman, J, The elements of statistical learning: data mining, inference and prediction, (2001), Springer Berlin · Zbl 0973.62007
[7] Huber, P, Robust estimation of a location parameter, Annals of mathematical statistics, 35, 73-101, (1964) · Zbl 0136.39805
[8] Kwok, J.T, Linear dependency between ε and the input noise in ε-support vector regression, (), 405-410 · Zbl 1001.68888
[9] Mattera, D; Haykin, S, Support vector machines for dynamic reconstruction of a chaotic system, ()
[10] Muller, K; Smola, A; Ratsch, G; Schölkopf, B; Kohlmorgen, J; Vapnik, V, Using support vector machines for time series prediction, ()
[11] Schölkopf, B; Bartlett, P; Smola, A; Williamson, R, Support vector regression with automatic accuracy control, (), 111-116
[12] Schölkopf, B; Burges, J; Smola, A, Advances in kernel methods: support vector machine, (1999), MIT Press Cambridge, MA
[13] Schölkopf, B; Smola, A, Learning with kernels: support vector machines, regularization, and beyond, (2002), MIT Press Cambridge, MA
[14] Smola, A; Murata, N; Schölkopf, B; Muller, K, Asymptotically optimal choice of ε-loss for support vector machines, Proceedings of ICANN 1998, (1998)
[15] Smola, A., & Schölkopf, B (1998). A tutorial on support vector regression. NeuroCOLT Technical Report NC-TR-98-030, Royal Holloway College, University of London, UK.
[16] Vapnik, V, Statistical learning theory, (1998), Wiley New York
[17] Vapnik, V, The nature of statistical learning theory, (1999), Springer Berlin · Zbl 0928.68093
[18] Vapnik, V (2001). Personal communication.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.