×

Anomalies in the foundations of ridge regression. (English) Zbl 1206.62129

Summary: Errors persist in ridge regression, its foundations and its usage, as set forth by A. E. Hoerl and R. W. Kennard [Technometrics 12, 55–67 (1970; Zbl 0202.17205)] and elsewhere. Ridge estimators need not be minimizing, nor a prospective ridge parameter be admissible. Conventional estimators are not LaGrange’s solutions constrained to fixed lengths, as claimed, since such solutions are singular. Of a massive literature on estimation, prediction, cross – validation, choice of ridge parameter, and related issues, little emanates from constrained optimization to include inequality constraints. The problem traces to a misapplication of LaGrange’s principle, unrecognized singularities, and misplaced links between constraints and ridge parameters. Alternative principles, based on condition numbers, are seen to validate both conventional ridge and surrogate ridge regression to be defined. Numerical studies illustrate that ridge regression as practiced often exhibits pathologies it is intended to redress.

MSC:

62J07 Ridge regression; shrinkage estimators (Lasso)
62H12 Estimation in multivariate analysis
65C60 Computational problems in statistics (MSC2010)

Citations:

Zbl 0202.17205
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] DOI: 10.2307/1267500 · Zbl 0286.62044 · doi:10.2307/1267500
[2] Balakrishnan A.V., J. Contr. 1 pp 109– (1963)
[3] DOI: 10.2307/2285761 · Zbl 0329.62049 · doi:10.2307/2285761
[4] Belsley D.A., Regression Diagnostics (1980) · Zbl 0479.62056 · doi:10.1002/0471725153
[5] DOI: 10.2307/2286476 · Zbl 0372.62052 · doi:10.2307/2286476
[6] Bunke O., Math. Operationsforsch. u. Statist. 6 pp 697– (1975) · Zbl 0318.62051 · doi:10.1080/02331887508801249
[7] DOI: 10.2307/2335377 · Zbl 0309.62043 · doi:10.2307/2335377
[8] DOI: 10.2307/2286909 · Zbl 0366.62086 · doi:10.2307/2286909
[9] DOI: 10.2307/1266023 · Zbl 0124.35003 · doi:10.2307/1266023
[10] DOI: 10.2307/1268518 · Zbl 0461.62059 · doi:10.2307/1268518
[11] DOI: 10.2307/2529336 · Zbl 0328.62042 · doi:10.2307/2529336
[12] Hoerl A.E., Chem. Eng. Prog. 58 pp 54– (1962)
[13] Hoerl A.E., Chem. Eng. Progr. Symposium Series 60 pp 67– (1964)
[14] DOI: 10.2307/1267351 · Zbl 0202.17205 · doi:10.2307/1267351
[15] DOI: 10.1080/03610927508827232 · Zbl 0296.62062 · doi:10.1080/03610927508827232
[16] Horn R.A., Matrix Analysis (1985) · Zbl 0576.15001 · doi:10.1017/CBO9780511810817
[17] Levenberg K., Quart. Appl. Math. 2 pp 164– (1944)
[18] DOI: 10.2307/1267380 · Zbl 0269.62061 · doi:10.2307/1267380
[19] DOI: 10.2307/1267205 · Zbl 0205.46102 · doi:10.2307/1267205
[20] DOI: 10.2307/2683673 · Zbl 0361.62060 · doi:10.2307/2683673
[21] Marshall A.W., Inequalities: Theory of Majorization and Its Applications (1979) · Zbl 0437.26007
[22] Myers R.H., Classical and Modern Regression with Applications, 2. ed. (1990)
[23] DOI: 10.1007/BF01893398 · Zbl 0381.62057 · doi:10.1007/BF01893398
[24] DOI: 10.2307/2287389 · doi:10.2307/2287389
[25] DOI: 10.2307/2002065 · Zbl 0066.10102 · doi:10.2307/2002065
[26] Stoer J., Convexity and Optimization in Finite Dimensions I (1970) · Zbl 0203.52203 · doi:10.1007/978-3-642-46216-0
[27] Tibshirani R., J. R. Stat. Soc. Ser. B 58 pp 267– (1996)
[28] DOI: 10.1081/STA-200063266 · Zbl 1072.62057 · doi:10.1081/STA-200063266
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.