Pötscher, Benedikt M. and Leeb, Hannes (2007): On the distribution of penalized maximum likelihood estimators: The LASSO, SCAD, and thresholding.
This is the latest version of this item.

PDF
MPRA_paper_14708.pdf Download (365kB)  Preview 
Abstract
We study the distributions of the LASSO, SCAD, and thresholding estimators, in finite samples and in the largesample limit. The asymptotic distributions are derived for both the case where the estimators are tuned to perform consistent model selection and for the case where the estimators are tuned to perform conservative model selection. Our findings complement those of Knight and Fu (2000) and Fan and Li (2001). We show that the distributions are typically highly nonnormal regardless of how the estimator is tuned, and that this property persists in large samples. The uniform convergence rate of these estimators is also obtained, and is shown to be slower than n^{1/2} in case the estimator is tuned to perform consistent model selection. An impossibility result regarding estimation of the estimators' distribution function is also provided.
Item Type:  MPRA Paper 

Institution:  University of Vienna 
Original Title:  On the distribution of penalized maximum likelihood estimators: The LASSO, SCAD, and thresholding. 
Language:  English 
Keywords:  Penalized maximum likelihood; LASSO; SCAD; thresholding; postmodelselection estimator; finitesample distribution; asymptotic distribution; oracle property; estimation of distribution; uniform consistency 
Subjects:  C  Mathematical and Quantitative Methods > C1  Econometric and Statistical Methods and Methodology: General > C13  Estimation: General C  Mathematical and Quantitative Methods > C2  Single Equation Models ; Single Variables 
Item ID:  14708 
Depositing User:  Benedikt Poetscher 
Date Deposited:  21. Apr 2009 00:12 
Last Modified:  12. Feb 2013 19:14 
References:  Bauer, P., Pötscher, B. M. & P. Hackl (1988): Model selection by multiple test procedures. Statistics 19, 3944. Beran, R. (1997): Diagnosing bootstrap success. Annals of the Institute of Statistical Mathematics 49, 124. Bruce, A. G. & H. Gao (1996): Understanding WaveShrink: Variance and bias estimation. Biometrika 83, 727745. Efron, B., Hastie, T., Johnstone, I. & R. Tibshirani (2004): Least angle regression. Annals of Statistics 32, 407499. Fan, J. & R. Li (2001): Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association 96, 13481360. Frank, I. E. & J. H. Friedman (1993): A statistical view of some chemometrics regression tools (with discussion). Technometrics 35, 109148. Kabaila, P. (1995): The effect of model selection on confidence regions and prediction regions. Econometric Theory 11, 537549. Knight, K. & W. Fu (2000): Asymptotics for lassotype estimators. Annals of Statistics 28, 13561378. Knight, K. (2008): Shrinkage estimation for nearlysingular designs. Econometric Theory 24, forthcoming. Kulperger, R. J. & S. E. Ahmed (1992): A bootstrap theorem for a preliminary test estimator. Communications in Statistics: Theory and Methods 21, 20712082. Judge, G. G. & M. E. Bock (1978): The Statistical Implications of Pretest and SteinRule Estimators in Econometrics. NorthHolland. Leeb, H. & B. M. Pötscher (2003): The finitesample distribution of postmodelselection estimators and uniform versus nonuniform approximations. Econometric Theory 19, 100142. Leeb, H. & B. M. Pötscher (2005): Model selection and inference: Facts and fiction. Econometric Theory 21, 2159. Leeb, H. & B. M. Pötscher (2006a): Performance limits for estimators of the risk or distribution of shrinkagetype estimators, and some general lower riskbound results. Econometric Theory 22, 6997. (Corrigendum: ibidem, 24, 581583). Leeb, H. & B. M. Pötscher (2006b): Can one estimate the conditional distribution of postmodelselection estimators? Annals of Statistics 34, 25542591. Leeb, H. & B. M. Pötscher (2008a): Sparse estimators and the oracle property, or the return of Hodges' estimator. Journal of Econometrics 142, 201211. Leeb, H. & B. M. Pötscher (2008b): Can one estimate the unconditional distribution of postmodelselection estimators? Econometric Theory 24, 338376. Lehmann, E. L. & G. Casella (1998): Theory of Point Estimation. Springer Texts in Statistics. New York: SpringerVerlag. Pötscher, B. M. (1991): Effects of model selection on inference. Econometric Theory 7, 163185. Pötscher, B. M. (2006): The distribution of model averaging estimators and an impossibility result regarding its estimation. IMS Lecture NotesMonograph Series 52, 113129. Pötscher, B. M. & U. Schneider (2009): On the distribution of the adaptive lasso estimator. Journal of Statistical Planning and Inference, forthcoming, doi:10.1016/j.jspi.2009.01.003. Samworth, R. (2003): A note on methods of restoring consistency of the bootstrap. Biometrika 90, 985990. Sen, P. K. (1979): Asymptotic properties of maximum likelihood estimators based on conditional specification. Annals of Statistics 7, 10191033. Tibshirani, R. (1996): Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B 58, 267288. Zhao, P. & B. Yu (2006): On model selection consistency of lasso. Journal of Machine Learning Research 7, 25412563. Zou, H. (2006): The adaptive lasso and its oracle properties. Journal of the American Statistical Association 101, 14181429. 
URI:  https://mpra.ub.unimuenchen.de/id/eprint/14708 
Available Versions of this Item

On the distribution of penalized maximum likelihood estimators: The LASSO, SCAD, and thresholding. (deposited 06. Nov 2007)
 On the distribution of penalized maximum likelihood estimators: The LASSO, SCAD, and thresholding. (deposited 21. Apr 2009 00:12) [Currently Displayed]