×

Model averaging estimator in ridge regression and its large sample properties. (English) Zbl 1461.62127

Summary: In linear regression, when the covariates are highly collinear, ridge regression has become the standard treatment. The choice of ridge parameter plays a central role in ridge regression. In this paper, instead of ending up with a single ridge parameter, we consider a model averaging method to combine multiple ridge estimators with \(M_n\) different ridge parameters, where \(M_n\) can go to infinity with sample size \(n\). We show that when the fitting model is correctly specified, the resulting model averaging estimator is \(n^{1/2}\)-consistent. When the fitting model is misspecified, the asymptotic optimality of the model averaging estimator is also established rigorously. The results of simulation studies and our case study concerning the urbanization level of Chinese ethnic areas demonstrate the usefulness of the model averaging method.

MSC:

62J07 Ridge regression; shrinkage estimators (Lasso)
62F12 Asymptotic properties of parametric estimators

Software:

AS 223
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Buckland, ST; Burnham, KP; Augustin, NH, Model selection: an integral part of inference, Biometrics, 53, 603-618 (1997) · Zbl 0885.62118
[2] Chen, X.; Zou, G.; Zhang, X., Frequentist model averaging for linear mixed-effects models, Front Math China, 8, 497-515 (2013) · Zbl 1273.62154
[3] Claeskens, G.; Croux, C.; van Kerckhoven, J., Variable selection for logistic regression using a prediction-focused information criterion, Biometrics, 62, 972-979 (2006) · Zbl 1116.62073
[4] Clue, E.; Vineis, P.; De Iorio, M., Significance testing in ridge regression for genetic data, BMC Bioinform, 12, 372 (2011)
[5] Dempster, A.; Schatzoff, M.; Wermuth, N., A simulation study of alternatives to ordinary least squares, J Am Stat Assoc, 70, 77-106 (1975) · Zbl 0366.62086
[6] Flynn, CJ; Hurvich, CM; Simonoff, JS, Efficiency for regularization parameter selection in penalized likelihood estimation of misspecified models, J Am Stat Assoc, 108, 1031-1043 (2013) · Zbl 06224985
[7] Gao, Y.; Zhang, X.; Wang, S.; Zou, G., Model averaging based on leave-subject-out cross-validation, J Econ, 192, 139-151 (2016) · Zbl 1419.62084
[8] Ghosh, S.; Yuan, Z., An improved model averaging scheme for logistic regression, J Multivar Anal, 100, 1670-1681 (2009) · Zbl 1163.62029
[9] Golub, G.; Heath, M.; Wahba, G., Generalized cross-validation as a method for choosing a good ridge parameter, Technometrics, 21, 215-223 (1979) · Zbl 0461.62059
[10] Hansen, BE, Least squares model averaging, Econometrica, 75, 1175-1189 (2007) · Zbl 1133.91051
[11] Hansen, BE; Racine, J., Jacknife model averaging, J Econ, 167, 38-46 (2012) · Zbl 1441.62721
[12] Hoerl, A.; Kennard, R., Ridge regression, biased estimation for nonorthogonal problems, Technometrics, 12, 55-67 (1970) · Zbl 0202.17205
[13] Lee, T., Algorithm as 223: optimum ridge parameter selection, J R Stat Soc C, 36, 112-118 (1987) · Zbl 0613.62092
[14] Leung, G.; Barron, AR, Information theory and mixing least-squares regressions, IEEE Trans Inf Theory, 52, 3396-3410 (2006) · Zbl 1309.94051
[15] Liu, Q.; Okui, R., Heteroskedasticity-robust Cp model averaging, Econ J, 16, 463-472 (2013)
[16] Lu, X.; Su, L., Jackknife model averaging for quantile regressions, J Econ, 188, 40-58 (2015) · Zbl 1337.62080
[17] Magnus, J.; De Luca, G., Weighted-average least squares (WALS), a survey, J Econ Surv, 30, 117-148 (2016)
[18] Moral-Benito, E., Model averaging in economics: an overview, J Econ, 29, 46-75 (2015)
[19] Nordberg, L., A procedure for determination of a good ridge parameter in linear regression, Commun Stat Simul Comput, 11, 285-309 (1982) · Zbl 0514.62078
[20] Rao, CR, Linear statistical inference and its applications (1973), New York: Wiley, New York · Zbl 0256.62002
[21] Schomaker, M., Shrinkage averaging estimation, Stat Pap, 53, 4, 1015-1034 (2012) · Zbl 1254.62082
[22] Schomaker, M.; Wan, ATK; Heumann, C., Frequentist model averaging with missing observations, Comput Stat Data Anal, 54, 3336-3347 (2010) · Zbl 1284.62063
[23] Wan, ATK; Zhang, X.; Zou, G., Least squares model averaging by Mallows criterion, J Econ, 156, 277-283 (2010) · Zbl 1431.62291
[24] Wang, H.; Zhang, X.; Zou, G., Frequentist model averaging estimation: a review, J Syst Sci Complex, 22, 732-748 (2009) · Zbl 1300.93164
[25] Yu, Y.; Thurston, S.; Hauser, R.; Liang, H., Model averaging procedure for partially linear single-index models, J Stat Plan Inference, 143, 2160-2170 (2013) · Zbl 1278.62112
[26] Yuan, Z.; Yang, Y., Combining linear regression models: when and how?, J Am Stat Assoc, 100, 1202-1214 (2005) · Zbl 1117.62454
[27] Zhang, X., Consistency of model averaging estimators, Econ Lett, 130, 120-123 (2015) · Zbl 1321.62022
[28] Zhang, X.; Liang, H., Focused information criterion and model averaging for generalized additive partial linear models, Ann Stat, 39, 174-200 (2011) · Zbl 1209.62088
[29] Zhang X, Wang W (2017) Optimal model averaging estimation for partially linear models. Stat Sin (forthcoming)
[30] Zhang, X.; Wan, ATK; Zhou, SZ, Focused information criteria, model selection and model averaging in a Tobit model with a non-zero threshold, J Busi Econ Stat, 30, 132-142 (2012)
[31] Zhang, X.; Wan, A.; Zou, G., Model averaging by jackknife criterion in models with dependent data, J Econ, 174, 82-94 (2013) · Zbl 1283.62059
[32] Zhang, X.; Zou, G.; Liang, H., Model averaging and weight choice in linear mixed-effects models, Biometrika, 101, 205-218 (2014) · Zbl 1285.62077
[33] Zhang, X.; Zou, G.; Carroll, R., Model averaging based on Kullback-Leibler distance, Stat Sin, 25, 1583-1598 (2015) · Zbl 1377.62089
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.