Can One Estimate the Unconditional Distribution of Post-Model-Selection Estimators ?

Leeb, Hannes and Pötscher, Benedikt M. (2005): Can One Estimate the Unconditional Distribution of Post-Model-Selection Estimators ?

Abstract

We consider the problem of estimating the unconditional distribution of a post-model-selection estimator. The notion of a post-model-selection estimator here refers to the combined procedure resulting from first selecting a model (e.g., by a model selection criterion like AIC or by a hypothesis testing procedure) and then estimating the parameters in the selected model (e.g., by least-squares or maximum likelihood), all based on the same data set. We show that it is impossible to estimate the unconditional distribution with reasonable accuracy even asymptotically. In particular, we show that no estimator for this distribution can be uniformly consistent (not even locally). This follows as a corollary to (local) minimax lower bounds on the performance of estimators for the distribution; performance is here measured by the probability that the estimation error exceeds a given threshold. These lower bounds are shown to approach 1/2 or even 1 in large samples, depending on the situation considered. Similar impossibility results are also obtained for the distribution of linear functions (e.g., predictors) of the post-model-selection estimator.

• Item Type: MPRA Paper
• Language: English
• Keywords: Inference after model selection; Post-model-selection estimator; Pre-test estimator; Selection of regressors; Akaike's information criterion AIC; Thresholding; Model uncertainty; Consistency; Uniform consistency; Lower risk bound
• Subjects: C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C51 - Model Construction and Estimation; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables > C20 - General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C12 - Hypothesis Testing: General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C13 - Estimation: General; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C52 - Model Evaluation, Validation, and Selection

References

Ahmed, S. E. & A. K. Basu (2000): Least squares, preliminary test and Stein-type estimation in general vector AR(p) models. Statistica Neerlandica 54, 47--66. Bauer, P., Pötscher, B. M. & P. Hackl (1988): Model selection by multiple test procedures. Statistics 19, 39--44. Billingsley, P. (1995): Probability and Measure, (3rd ed.), Wiley. Brownstone, D. (1990): Bootstrapping improved estimators for linear regression models. Journal of Econometrics 44, 171--187. Danilov, D. L. & J. R. Magnus (2004): On the harm that ignoring pre-testing can cause. Journal of Econometrics 122, 27--46. Dijkstra, T. K. & J. H. Veldkamp (1988): `Data-driven selection of regressors and the bootstrap'. Lecture Notes in Economics and Mathematical Systems 307, 17--38. Dukić, V. M. & E. A Peña (2005): Variance estimation in a model with gaussian submodels. Journal of the American Statistical Association 100, 296-309. Freedman, D. A., Navidi, W. & S. C. Peters (1988): `On the impact of variable selection in fitting regression equations'. Lecture Notes in Economics and Mathematical Systems 307, 1--16. Hansen, P. R. (2003): Regression analysis with many specifications: a bootstrap method for robust inference. Working Paper, Department of Economics, Brown University. Hjort, N. L. & G. Claeskens (2003): Frequentist model average estimators. Journal of the American Statistical Association 98, 879--899. Kabaila, P. (1995): The effect of model selection on confidence regions and prediction regions. Econometric Theory 11, 537--549. Kapetanios, G. (2001): Incorporating lag order selection uncertainty in parameter inference for AR models. Economics Letters 72, 137--144. Kilian, L. (1998): Accounting for lag order uncertainty in autoregressions: the endogenous lag order bootstrap algorithm. Journal of Time Series Analysis 19, 531--548. Knight, K. (1999): Epi-convergence in distribution and stochastic equi-semicontinuity. Working Paper, Department of Statistics, University of Toronto. Kulperger, R. J. & S. E. Ahmed (1992): A bootstrap theorem for a preliminary test estimator. Communications in Statistics: Theory and Methods 21, 2071--2082. Leeb, H. (2002): On a differential equation with advanced and retarded arguments. Communications on Applied Nonlinear Analysis 9, 77--86. Leeb, H. (2005): The distribution of a linear predictor after model selection: conditional finite-sample distributions and asymptotic approximations. Journal of Statistical Planning and Inference 134, 64--89. Leeb, H. (2003): The distribution of a linear predictor after model selection: unconditional finite-sample distributions and asymptotic approximations. IMS Lecture Notes-Monograph Series 49, 291-311. Leeb, H. & B. M. Pötscher (2003): The finite-sample distribution of post-model-selection estimators and uniform versus nonuniform approximations. Econometric Theory 19, 100--142. Leeb, H. & B. M. Pötscher (2005a): Model Selection and Inference: Facts and Fiction. Econometric Theory 21, 21--59. Leeb, H. & B. M. Pötscher (2005b): Can one estimate the conditional distribution of post-model-selection estimators? Working Paper, Department of Statistics, University of Vienna. Leeb, H. & B. M. Pötscher (2006a): Performance limits for estimators of the risk or distribution of shrinkage-type estimators, and some general lower risk bound results. Econometric Theory 22, 69-97. (Corrigendum, ibid., forthcoming.) Leeb, H. & B. M. Pötscher (2006b): Can one estimate the conditional distribution of post-model-selection estimators? Annals of Statistics 34, 2554-2591. Lehmann, E.L. & G. Casella (1998): Theory of Point Estimation, 2nd edition, Springer Texts in Statistics. Springer-Verlag. Nickl, R. (2003): Asymptotic Distribution Theory of Post-Model-Selection Estimators. Masters Thesis, Department of Statistics, University of Vienna. Pötscher, B. M. (1991): Effects of model selection on inference. Econometric Theory 7, 163--185. Pötscher, B. M. (1995): Comment on `The effect of model selection on confidence regions and prediction regions' by P. Kabaila. Econometric Theory 11, 550--559. Pötscher, B. M. & A. J. Novak (1998): The distribution of estimators after model selection: large and small sample results. Journal of Statistical Computation and Simulation 60, 19--56. Rao, C. R. & Y. Wu (2001): `On model selection,' IMS Lecture Notes Monograph Series 38, 1--57. Sen, P. K. (1979): Asymptotic properties of maximum likelihood estimators based on conditional specification. Annals of Statistics 7, 1019--1033. Sen P. K. & A. K. M. E. Saleh (1987): On preliminary test and shrinkage M-estimation in linear models. Annals of Statistics 15, 1580--1592. van der Vaart, A. W. (1998): Asymptotic Statistics. Cambridge University Press.