Distributional results for thresholding estimators in high-dimensional Gaussian regression models

Pötscher, Benedikt M. and Schneider, Ulrike (2011): Distributional results for thresholding estimators in high-dimensional Gaussian regression models.

Abstract

We study the distribution of hard-, soft-, and adaptive soft-thresholding estimators within a linear regression model where the number of parameters k can depend on sample size n and may diverge with n. In addition to the case of known error-variance, we define and study versions of the estimators when the error-variance is unknown. We derive the finite-sample distribution of each estimator and study its behavior in the large-sample limit, also investigating the effects of having to estimate the variance when the degrees of freedom n-k does not tend to infinity or tends to infinity very slowly. Our analysis encompasses both the case where the estimators are tuned to perform consistent variable selection and the case where the estimators are tuned to perform conservative variable selection. Furthermore, we discuss consistency, uniform consistency and derive the minimax rate under either type of tuning.

• Item Type: MPRA Paper
• Language: English
• Keywords: Thresholding, Lasso, adaptive Lasso, penalized maximum likelihood, variable selection, finite-sample distribution, asymptotic distribution, variance estimation, minimax rate, high-dimensional model, oracle property
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C13 - Estimation: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C20 - General; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C52 - Model Evaluation, Validation, and Selection

References

Alliney, S. & S. A. Ruzinsky (1994): An algorithm for the minimization of mixed l₁ and l₂ norms with applications to Bayesian estimation. IEEE Transactions on Signal Processing 42, 618-627.

Bauer, P., Pötscher, B. M. & P. Hackl (1988): Model selection by multiple test procedures. Statistics 19, 39--44.

Donoho, D. L., Johnstone, I. M., Kerkyacharian, G., D. Picard (1995): Wavelet shrinkage: asymptopia? With discussion and a reply by the authors. Journal of the Royal Statistical Society Series B 57, 301--369.

Fan, J. & R. Li (2001): Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association 96, 1348-1360.

Fan, J. & H. Peng (2004): Nonconcave penalized likelihood with a diverging number of parameters. Annals of Statistics 32, 928--961.

Feller, W. (1957): An Introduction to Probability Theory and Its Applications, Volume 1. 2nd ed., Wiley, New York.

Frank, I. E. & J. H. Friedman (1993): A statistical view of some chemometrics regression tools (with discussion). Technometrics 35, 109-148.

Ibragimov, I. A. (1956): On the composition of unimodal distributions. Theory of Probability and its Applications 1, 255-260.

Knight, K. & W. Fu (2000): Asymptotics for lasso-type estimators. Annals of Statistics 28, 1356-1378.

Leeb, H. & B. M. Pötscher (2003): The finite-sample distribution of post-model-selection estimators and uniform versus nonuniform approximations. Econometric Theory 19, 100--142.

Leeb, H. & B. M. Pötscher (2005): Model selection and inference: facts and fiction. Econometric Theory 21, 21--59.

Leeb, H. & B. M. Pötscher (2008): Sparse estimators and the oracle property, or the return of Hodges' estimator. Journal of Econometrics 142, 201-211.

Pötscher, B. M. (1991): Effects of model selection on inference. Econometric Theory 7, 163--185.

Pötscher, B. M. (2006): The distribution of model averaging estimators and an impossibility result regarding its estimation. IMS Lecture Notes-Monograph Series 52, 113--129.

Pötscher, B. M. & H. Leeb (2009): On the distribution of penalized maximum likelihood estimators: the LASSO, SCAD, and thresholding. Journal of Multivariate Analysis 100, 2065-2082.

Pötscher, B. M. & U. Schneider (2009): On the distribution of the adaptive LASSO estimator. Journal of Statistical Planning and Inference 139, 2775-2790.

Pötscher, B. M. & U. Schneider (2010): Confidence sets based on penalized maximum likelihood estimators in Gaussian regression. Electronic Journal of Statistics 10, 334-360.

Sen, P. K. (1979): Asymptotic properties of maximum likelihood estimators based on conditional specification. Annals of Statistics 7, 1019-1033.

Tibshirani, R. (1996): Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B 58, 267-288.

Zhang, C.-H. (2010): Nearly unbiased variable selection under minimax concave penalty. Annals of Statistics 38, 894-942.

Zou, H. (2006): The adaptive lasso and its oracle properties. Journal of the American Statistical Association 101, 1418-1429.