On the distribution of the adaptive LASSO estimator

Pötscher, Benedikt M. and Schneider, Ulrike (2007): On the distribution of the adaptive LASSO estimator.

Abstract

We study the distribution of the adaptive LASSO estimator (Zou (2006)) in finite samples as well as in the large-sample limit. The large-sample distributions are derived both for the case where the adaptive LASSO estimator is tuned to perform conservative model selection as well as for the case where the tuning results in consistent model selection. We show that the finite-sample as well as the large-sample distributions are typically highly non-normal, regardless of the choice of the tuning parameter. The uniform convergence rate is also obtained, and is shown to be slower than n^{-1/2} in case the estimator is tuned to perform consistent model selection. In particular, these results question the statistical relevance of the `oracle' property of the adaptive LASSO estimator established in Zou 2006). Moreover, we also provide an impossibility result regarding the estimation of the distribution function of the adaptive LASSO estimator. The theoretical results, which are obtained for a regression model with orthogonal design, are complemented by a Monte Carlo study using non-orthogonal regressors.

• Item Type: MPRA Paper
• Language: English
• Keywords: Penalized maximum likelihood, LASSO, adaptive LASSO, nonnegative garotte, finite-sample distribution, asymptotic distribution, oracle property, estimation of distribution, uniform consistency
• Subjects: C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C51 - Model Construction and Estimation; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C13 - Estimation: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C20 - General; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C52 - Model Evaluation, Validation, and Selection

References

\begin{thebibliography}{34} \expandafter\ifx\csname natexlab\endcsname\relax\def\natexlab#1{#1}\fi \expandafter\ifx\csname url\endcsname\relax \def\url#1{\texttt{#1}}\fi \expandafter\ifx\csname urlprefix\endcsname\relax\def\urlprefix{URL }\fi \providecommand{\eprint}[2][]{\url{#2}}

\bibitem[{Bauer et~al.(1988)Bauer, P\"{o}tscher \& Hackl}]{bauetal88} \textsc{Bauer, P.}, \textsc{P\"{o}tscher, B.~M.} \& \textsc{Hackl, P.} (1988). \newblock Model selection by multiple test procedures. \newblock \textit{Statistics} 19 39--44.

\bibitem[{Breiman(1995)}]{brei95} \textsc{Breiman, L.} (1995). \newblock Better subset regression using the nonnegative garotte. \newblock \textit{Technometrics} 37 373--384.

\bibitem[{Bunea(2004)}]{bun04} \textsc{Bunea, F.} (2004). \newblock Consistent covariate selection and post model selection inference in semiparametric regression. \newblock \textit{Annals of Statistics} 32 898--927.

\bibitem[{Bunea \& McKeague(2005)}]{bunmck05} \textsc{Bunea, F.} \& \textsc{McKeague, I.~W.} (2005). \newblock Covariate selection for semiparametric hazard function regression models. \newblock \textit{Journal of Multivariate Analysis} 92 186--204.

\bibitem[{Efron et~al.(2004)Efron, Hastie, Johnstone \& Tibshirani}]{efretal04} \textsc{Efron, B.}, \textsc{Hastie, T.}, \textsc{Johnstone, I.} \& \textsc{Tibshirani, R.} (2004). \newblock Least angle regression. \newblock \textit{Annals of Statistics} 32 407--499.

\bibitem[{Fan \& Li(2001)}]{fanli01} \textsc{Fan, J.} \& \textsc{Li, R.} (2001). \newblock Variable selection via nonconcave penalized likelihood and its oracle properties. \newblock \textit{Journal of the American Statistical Association} 96 1348--1360.

\bibitem[{Fan \& Li(2002)}]{fanli02} \textsc{Fan, J.} \& \textsc{Li, R.} (2002). \newblock Variable selection for {C}ox's proportional hazards model and frailty model. \newblock \textit{Annals of Statistics} 30 74--99.

\bibitem[{Fan \& Li(2004)}]{fanli04} \textsc{Fan, J.} \& \textsc{Li, R.} (2004). \newblock New estimation and model selection procedures for semiparametric modeling in longitudinal data analysis. \newblock \textit{Journal of the American Statistical Association} 99 710--723.

\bibitem[{Frank \& Friedman(1993)}]{frafri93} \textsc{Frank, I.~E.} \& \textsc{Friedman, J.~H.} (1993). \newblock A statistical view of some chemometrics regression tools (with discussion). \newblock \textit{Technometrics} 35 109--148.

\bibitem[{Johnson et~al.(2008)Johnson, Lin \& Zeng}]{johetal08} \textsc{Johnson, B.}, \textsc{Lin, D.} \& \textsc{Zeng, D.} (2008). \newblock Penalized estimating functions and variable selection in semiparametric regression models. \newblock \textit{Journal of the American Statistical Association} 103 672--680.

\bibitem[{Kabaila(1995)}]{kab95} \textsc{Kabaila, P.} (1995). \newblock The effect of model selection on confidence regions and prediction regions. \newblock \textit{Econometric Theory} 11 537--549.

\bibitem[{Knight \& Fu(2000)}]{knifu00} \textsc{Knight, K.} \& \textsc{Fu, W.} (2000). \newblock Asymptotics of lasso-type estimators. \newblock \textit{Annals of Statistics} 28 1356--1378.

\bibitem[{Leeb \& P\"otscher(2003)}]{leepoe03} \textsc{Leeb, H.} \& \textsc{P\"otscher, B.~M.} (2003). \newblock The finite-sample distribution of post-model-selection estimators and uniform versus nonuniform approximations. \newblock \textit{Econometric Theory} 19 100--142.

\bibitem[{Leeb \& P\"otscher(2005)}]{leepoe05} \textsc{Leeb, H.} \& \textsc{P\"otscher, B.~M.} (2005). \newblock Model selection and inference: Facts and fiction. \newblock \textit{Econometric Theory} 21 21--59.

\bibitem[{Leeb \& P\"otscher(2006{\natexlab{a}})}]{leepoe06a} \textsc{Leeb, H.} \& \textsc{P\"otscher, B.~M.} (2006{\natexlab{a}}). \newblock Can one estimate the conditional distribution of post-model-selection estimators? \newblock \textit{Annals of Statistics} 34 2554--2591.

\bibitem[{Leeb \& P\"otscher(2006{\natexlab{b}})}]{leepoe06b} \textsc{Leeb, H.} \& \textsc{P\"otscher, B.~M.} (2006{\natexlab{b}}). \newblock Performance limits for estimators of the risk or distribution of shrinkage-type estimators, and some general lower risk-bound results. \newblock \textit{Econometric Theory} 22 69--97. \newblock (Corrections: ibidem, 24, 581-583).

\bibitem[{Leeb \& P\"otscher(2008{\natexlab{a}})}]{leepoe08b} \textsc{Leeb, H.} \& \textsc{P\"otscher, B.~M.} (2008{\natexlab{a}}). \newblock Can one estimate the unconditional distribution of post-model-selection estimators? \newblock \textit{Econometric Theory} 24 338--376.

\bibitem[{Leeb \& P\"otscher(2008{\natexlab{b}})}]{leepoe08a} \textsc{Leeb, H.} \& \textsc{P\"otscher, B.~M.} (2008{\natexlab{b}}). \newblock Sparse estimators and the oracle pro\-per\-ty, or the return of {H}odges' estimator. \newblock \textit{Journal of Econometrics} 142 201--211.

\bibitem[{Leng et~al.(2006)Leng, Lin \& Wahba}]{leliwa06} \textsc{Leng, C.}, \textsc{Lin, Y.} \& \textsc{Wahba, G.} (2006). \newblock A note on the lasso and related procedures in model selection. \newblock \textit{Statistica Sinica} 16 1273--1284.

\bibitem[{Li \& Liang(2007)}]{lilia07} \textsc{Li, R.} \& \textsc{Liang, H.} (2007). \newblock Variable selection in semiparametric regression modeling. \newblock \textit{Annals of Statistics} 36 261--286.

\bibitem[{P\"otscher(2006)}]{poe06} \textsc{P\"otscher, B.~M.} (2006). \newblock The distribution of model averaging estimators and an impossibility result regarding its estimation. \newblock \textit{IMS Lecture Notes - Monograph Series} 52 113--129.

\bibitem[{P\"otscher(2007)}]{poe07} \textsc{P\"otscher, B.~M.} (2007). \newblock Confidence sets based on sparse estimators are necessarily large. \newblock \textit{Manuscript} ArXiv:0711.1036.

\bibitem[{P\"otscher \& Leeb(2007)}]{poelee07} \textsc{P\"otscher, B.~M.} \& \textsc{Leeb, H.} (2007). \newblock On the distribution of penalized maximum likelihood estimators: The {LASSO}, {SCAD}, and thresholding. \newblock \textit{Manuscript} ArXiv:0711.0660.

\bibitem[{Tibshirani(1996)}]{tib96} \textsc{Tibshirani, R.} (1996). \newblock Regression shrinkage and selection via the lasso. \newblock \textit{Journal of the Royal Statistical Society Series B} 58 267--288.

\bibitem[{Wang \& Leng(2007)}]{wanlen07} \textsc{Wang, H.} \& \textsc{Leng, C.} (2007). \newblock Unified lasso estimation by least squares approximation. \newblock \textit{Journal of the American Statistical Association} 102 1039--1048.

\bibitem[{Wang et~al.(2007{\natexlab{a}})Wang, Li \& Jiang}]{wanglijia07} \textsc{Wang, H.}, \textsc{Li, G.} \& \textsc{Jiang, G.} (2007). \newblock Robust regression shrinkage and consistent variable selection through the {LAD}-lasso. \newblock \textit{Journal of Business and Economic Statistics} 25 347--355.

\bibitem[{Wang et~al.(2007{\natexlab{b}})Wang, Li \& Tsai}]{wanglitsa07} \textsc{Wang, H.}, \textsc{Li, G.} \& \textsc{Tsai, C.~L.} (2007). \newblock Regression coefficient and autoregressive order shrinkage and selection via the lasso. \newblock \textit{Journal of the Royal Statistical Society Series B} 69 63--78.

\bibitem[{Wang et~al.(2007{\natexlab{c}})Wang, Li \& Tsai}]{wanrlitsa07} \textsc{Wang, H.}, \textsc{Li, R.} \& \textsc{Tsai, C.~L.} (2007). \newblock Tuning parameter selectors for the smooth\-ly clipped absolute deviation method. \newblock \textit{Biometrika} 94 553--568.

\bibitem[{Yang(2005)}]{yan05} \textsc{Yang, Y.} (2005). \newblock Can the strengths of {AIC} and {BIC} be shared? {A} conflict between model indentification and regression estimation. \newblock \textit{Biometrika} 92 937--950.

\bibitem[{Yuan \& Lin(2007)}]{yualin07} \textsc{Yuan, M.} \& \textsc{Lin, Y.} (2007). \newblock Model selection and estimation in the gaussian graphical model. \newblock \textit{Biometrika} 94 19--35.

\bibitem[{Zhang \& Lu(2007)}]{zhalu07} \textsc{Zhang, H.~H.} \& \textsc{Lu, W.} (2007). \newblock Adaptive lasso for {C}ox's proportional hazards model. \newblock \textit{Biometrika} 94 691--703.

\bibitem[{Zou(2006)}]{zou06} \textsc{Zou, H.} (2006). \newblock The adaptive lasso and its oracle properties. \newblock \textit{Journal of the American Statistical Association} 101 1418--1429.

\bibitem[{Zou \& Li(2008)}]{zouli08} \textsc{Zou, H.} \& \textsc{Li, R.} (2008). \newblock One-step sparse estimates in nonconcave penalized likelihood models. \newblock \textit{Annals of Statistics} 36 1509--1533.

\bibitem[{Zou \& Yuan(2008)}]{zouyua08} \textsc{Zou, H.} \& \textsc{Yuan, M.} (2008). \newblock Composite quantile regression and the oracle model selection theory. \newblock \textit{Annals of Statistics} 36 1108--1126.

\end{thebibliography}