Distributional results for thresholding estimators in high-dimensional Gaussian regression models. (English) Zbl 1271.62149

Summary: We study the distribution of hard-, soft-, and adaptive soft-thresholding estimators within a linear regression model where the number of parameters \(k\) can depend on sample size \(n\) and may diverge with \(n\). In addition to the case of known error-variance, we define and study versions of the estimators when the error-variance is unknown. We derive the finite-sample distribution of each estimator and study its behavior in the large-sample limit, also investigating the effects of having to estimate the variance when the degrees of freedom \(n-k\) does not tend to infinity or tends to infinity very slowly. Our analysis encompasses both the case where the estimators are tuned to perform consistent variable selection and the case where the estimators are tuned to perform conservative variable selection. Furthermore, we discuss consistency, uniform consistency and derive the uniform convergence rate under either type of tuning.


62J05 Linear regression; mixed models
62F10 Point estimation
62E15 Exact distribution theory in statistics
62E20 Asymptotic distribution theory in statistics
Full Text: DOI arXiv Euclid


[1] Alliney, S. and Ruzinsky, A. (1994). An Algorithm For the Minimization of Mixed, l 1 and l 2 Norms With Applications to Bayesian Estimation. IEEE Transactions on Signal Processing 42 618-627.
[2] Bauer, P., Pötscher, B. M. and Hackl, P. (1988). Model Selection by Multiple Test Procedures., Statistics 19 39-44. · Zbl 0644.62024 · doi:10.1080/02331888808802068
[3] Donoho, D. L., Johnstone, I. M., Kerkyacharian, G. and Picard, D. (1995). Wavelet Shrinkage: Asymptopia? With discussion and a reply by the authors., Journal of the Royal Statistical Society Series B 57 301-369. · Zbl 0827.62035
[4] Fan, J. and Li, R. (2001). Variable Selection Via Nonconcave Penalized Likelihood and Its Oracle Properties., Journal of the American Statistical Association 96 1348-1360. · Zbl 1073.62547 · doi:10.1198/016214501753382273
[5] Fan, J. and Peng, H. (2004). Nonconcave Penalized Likelihood With a Diverging Number of Parameters., Annals of Statistics 32 928-961. · Zbl 1092.62031 · doi:10.1214/009053604000000256
[6] Feller, W. (1957)., An Introduction to Probability Theory and Its Applications 1 . Wiley, New York. · Zbl 0077.12201
[7] Frank, I. E. and Friedman, J. H. (1993). A Statistical View of Some Chemometrics Regression Tools (with discussion)., Technometrics 35 109-148. · Zbl 0775.62288 · doi:10.2307/1269656
[8] Ibragimov, I. A. (1956). On the Composition of Unimodal Distributions., Theory of Probability and its Applications 1 255-260. · Zbl 0073.12501
[9] Knight, K. and Fu, W. (2000). Asymptotics of Lasso-Type Estimators., Annals of Statistics 28 1356-1378. · Zbl 1105.62357 · doi:10.1214/aos/1015957397
[10] Leeb, H. and Pötscher, B. M. (2003). The Finite-Sample Distribution of Post-Model-Selection Estimators and Uniform Versus Nonuniform Approximations., Econometric Theory 19 100-142. · Zbl 1032.62011 · doi:10.1017/S0266466603191050
[11] Leeb, H. and Pötscher, B. M. (2005). Model Selection and Inference: Facts and Fiction., Econometric Theory 21 21-59. · Zbl 1085.62004 · doi:10.1017/S0266466605050036
[12] Leeb, H. and Pötscher, B. M. (2008). Sparse Estimators and the Oracle Property, or the Return of Hodges’ Estimator., Journal of Econometrics 142 201-211. · Zbl 1418.62272 · doi:10.1016/j.jeconom.2007.05.017
[13] Pötscher, B. M. (1991). Effects of Model Selection on Inference., Econometric Theory 7 163-185. · Zbl 04504752 · doi:10.1017/S0266466600004382
[14] Pötscher, B. M. (2006). The Distribution of Model Averaging Estimators and an Impossibility Result Regarding its Estimation., IMS Lecture Notes - Monograph Series 52 113-129. · Zbl 1268.62066
[15] Pötscher, B. M. and Leeb, H. (2009). On the Distribution of Penalized Maximum Likelihood Estimators: The LASSO, SCAD, and Thresholding., Journal of Multivariate Analysis 100 2065-2082. · Zbl 1170.62046 · doi:10.1016/j.jmva.2009.06.010
[16] Pötscher, B. M. and Schneider, U. (2009). On the Distribution of the Adaptive LASSO Estimator., Journal of Statistical Planning and Inference 139 2775-2790. · Zbl 1162.62063 · doi:10.1016/j.jspi.2009.01.003
[17] Pötscher, B. M. and Schneider, U. (2010). Confidence Sets Based on Penalized Maximum Likelihood Estimators in Gaussian Regression., Electronic Journal of Statistics 4 334-360. · Zbl 1329.62156 · doi:10.1214/09-EJS523
[18] Sen, P. K. (1979). Asymptotic Properties of Maximum Likelihood Estimators Based on Conditional Specification., Annals of Statistics 7 1019-1033. · Zbl 0413.62020 · doi:10.1214/aos/1176344785
[19] Tibshirani, R. (1996). Regression Shrinkage and Selection Via the Lasso., Journal of the Royal Statistical Society Series B 58 267-288. · Zbl 0850.62538
[20] Zhang, C. (2010). Nearly Unbiased Variable Selection under Minimax Concave Penalty., Annals of Statistics 38 893-942. · Zbl 1183.62120 · doi:10.1214/09-AOS729
[21] Zou, H. (2006). The Adaptive Lasso and Its Oracle Properties., Journal of the American Statistical Association 101 1418-1429. · Zbl 1171.62326 · doi:10.1198/016214506000000735
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.