Distributional results for thresholding estimators in high-dimensional Gaussian regression models

Pötscher, Benedikt M. and Schneider, Ulrike (2011): Distributional results for thresholding estimators in high-dimensional Gaussian regression models.

Abstract

We study the distribution of hard-, soft-, and adaptive soft-thresholding estimators within a linear regression model where the number of parameters k can depend on sample size n and may diverge with n. In addition to the case of known error-variance, we define and study versions of the estimators when the error-variance is unknown. We derive the finite-sample distribution of each estimator and study its behavior in the large-sample limit, also investigating the effects of having to estimate the variance when the degrees of freedom n-k does not tend to infinity or tends to infinity very slowly. Our analysis encompasses both the case where the estimators are tuned to perform consistent model selection and the case where the estimators are tuned to perform conservative model selection. Furthermore, we discuss consistency, uniform consistency and derive the minimax rate under either type of tuning.

• Item Type: MPRA Paper
• Language: English
• Keywords: Thresholding, Lasso, adaptive Lasso, penalized maximum likelihood, finite-sample distribution, asymptotic distribution, variance estimation, minimax rate, high-dimensional model, oracle property
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C13 - Estimation: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C20 - General; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C52 - Model Evaluation, Validation, and Selection

References

Alliney, S. & S. A. Ruzinsky (1994): An algorithm for the minimization of mixed l₁ and l₂ norms with applications to Bayesian estimation. IEEE Transactions on Signal Processing 42, 618-627.

Bauer, P., Pötscher, B. M. & P. Hackl (1988): Model selection by multiple test procedures. Statistics 19, 39--44.

Donoho, D. L., Johnstone, I. M., Kerkyacharian, G., D. Picard (1995): Wavelet shrinkage: asymptopia? With discussion and a reply by the authors. Journal of the Royal Statistical Society Series B 57, 301--369.

Fan, J. & R. Li (2001): Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association 96, 1348-1360.

Fan, J. & H. Peng (2004): Nonconcave penalized likelihood with a diverging number of parameters. Annals of Statistics 32, 928--961.

Feller, W. (1957): An Introduction to Probability Theory and Its Applications, Volume 1. 2nd ed., Wiley, New York.

Frank, I. E. & J. H. Friedman (1993): A statistical view of some chemometrics regression tools (with discussion). Technometrics 35, 109-148.

Ibragimov, I. A. (1956): On the composition of unimodal distributions. Theory of Probability and its Applications 1, 255-260

Leeb, H. & B. M. Pötscher (2003): The finite-sample distribution of post-model-selection estimators and uniform versus nonuniform approximations. Econometric Theory 19, 100--142.

Leeb, H. & B. M. Pötscher (2005): Model selection and inference: facts and fiction. Econometric Theory 21, 21--59.

Leeb, H. & B. M. Pötscher (2008): Sparse estimators and the oracle property, or the return of Hodges' estimator. Journal of Econometrics 142, 201-211.

Knight, K. & W. Fu (2000): Asymptotics for lasso-type estimators. Annals of Statistics 28, 1356-1378.

Pötscher, B. M. & H. Leeb (2009): On the distribution of penalized maximum likelihood estimators: the LASSO, SCAD, and thresholding. Journal of Multivariate Analysis 100, 2065-2082.

Pötscher, B. M. & U. Schneider (2009): On the distribution of the adaptive LASSO estimator. Journal of Statistical Planning and Inference 139, 2775-2790.

Pötscher, B. M. & U. Schneider (2010): Confidence sets based on penalized maximum likelihood estimators in Gaussian regression. Electronic Journal of Statistics 10, 334-360.

Tibshirani, R. (1996): Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B 58, 267-288.

Zou, H. (2006): The adaptive lasso and its oracle properties. Journal of the American Statistical Association 101, 1418-1429.