On the distribution of penalized maximum likelihood estimators: The LASSO, SCAD, and thresholding.

Pötscher, Benedikt M. and Leeb, Hannes (2007): On the distribution of penalized maximum likelihood estimators: The LASSO, SCAD, and thresholding.

Abstract

We study the distributions of the LASSO, SCAD, and thresholding estimators, in finite samples and in the large-sample limit. The asymptotic distributions are derived for both the case where the estimators are tuned to perform consistent model selection and for the case where the estimators are tuned to perform conservative model selection. Our findings complement those of Knight and Fu (2000) and Fan and Li (2001). We show that the distributions are typically highly nonnormal regardless of how the estimator is tuned, and that this property persists in large samples. The uniform convergence rate of these estimators is also obtained, and is shown to be slower than n^{-1/2} in case the estimator is tuned to perform consistent model selection. An impossibility result regarding estimation of the estimators' distribution function is also provided.

• Item Type: MPRA Paper
• Institution: University of Vienna
• Language: English
• Keywords: Penalized maximum likelihood; LASSO; SCAD; thresholding; post-model-selection estimator; finite-sample distribution; asymptotic distribution; oracle property; estimation of distribution; uniform consistency
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C13 - Estimation: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables

References

Bauer, P., Pötscher, B. M. & P. Hackl (1988): Model selection by multiple test procedures. Statistics 19, 39--44.

Beran, R. (1997): Diagnosing bootstrap success. Annals of the Institute of Statistical Mathematics 49, 1-24.

Bruce, A. G. & H. Gao (1996): Understanding WaveShrink: Variance and bias estimation. Biometrika 83, 727-745.

Efron, B., Hastie, T., Johnstone, I. & R. Tibshirani (2004): Least angle regression. Annals of Statistics 32, 407--499.

Fan, J. & R. Li (2001): Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association 96, 1348-1360.

Frank, I. E. & J. H. Friedman (1993): A statistical view of some chemometrics regression tools (with discussion). Technometrics 35, 109-148.

Kabaila, P. (1995): The effect of model selection on confidence regions and prediction regions. Econometric Theory 11, 537--549.

Knight, K. & W. Fu (2000): Asymptotics for lasso-type estimators. Annals of Statistics 28, 1356-1378.

Knight, K. (2008): Shrinkage estimation for nearly-singular designs. Econometric Theory 24, forthcoming.

Kulperger, R. J. & S. E. Ahmed (1992): A bootstrap theorem for a preliminary test estimator. Communications in Statistics: Theory and Methods 21, 2071--2082.

Judge, G. G. & M. E. Bock (1978): The Statistical Implications of Pre-test and Stein-Rule Estimators in Econometrics. North-Holland.

Leeb, H. & B. M. Pötscher (2003): The finite-sample distribution of post-model-selection estimators and uniform versus nonuniform approximations. Econometric Theory 19, 100--142.

Leeb, H. & B. M. Pötscher (2005): Model selection and inference: Facts and fiction. Econometric Theory 21, 21--59.

Leeb, H. & B. M. Pötscher (2006a): Performance limits for estimators of the risk or distribution of shrinkage-type estimators, and some general lower risk-bound results. Econometric Theory 22, 69--97. (Corrigendum: ibidem, 24, 581-583).

Leeb, H. & B. M. Pötscher (2006b): Can one estimate the conditional distribution of post-model-selection estimators? Annals of Statistics 34, 2554-2591.

Leeb, H. & B. M. Pötscher (2008a): Sparse estimators and the oracle property, or the return of Hodges' estimator. Journal of Econometrics 142, 201-211.

Leeb, H. & B. M. Pötscher (2008b): Can one estimate the unconditional distribution of post-model-selection estimators? Econometric Theory 24, 338-376.

Lehmann, E. L. & G. Casella (1998): Theory of Point Estimation. Springer Texts in Statistics. New York: Springer-Verlag.

Pötscher, B. M. (1991): Effects of model selection on inference. Econometric Theory 7, 163--185.

Pötscher, B. M. (2006): The distribution of model averaging estimators and an impossibility result regarding its estimation. IMS Lecture Notes--Monograph Series 52, 113-129.

Pötscher, B. M. & U. Schneider (2009): On the distribution of the adaptive lasso estimator. Journal of Statistical Planning and Inference, forthcoming, doi:10.1016/j.jspi.2009.01.003.

Samworth, R. (2003): A note on methods of restoring consistency of the bootstrap. Biometrika 90, 985--990.

Sen, P. K. (1979): Asymptotic properties of maximum likelihood estimators based on conditional specification. Annals of Statistics 7, 1019-1033.

Tibshirani, R. (1996): Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B 58, 267-288.

Zhao, P. & B. Yu (2006): On model selection consistency of lasso. Journal of Machine Learning Research 7, 2541-2563.

Zou, H. (2006): The adaptive lasso and its oracle properties. Journal of the American Statistical Association 101, 1418-1429.