Multiple hypothesis testing of market risk forecasting models

esposito, francesco paolo and cummins, mark (2015): Multiple hypothesis testing of market risk forecasting models. Forthcoming in: Journal of Forecasting

Abstract

Extending previous risk model backtesting literature, we construct multiple hypothesis testing (MHT) with the stationary bootstrap. We conduct multiple tests which control for the generalized confidence level and employ the bootstrap MHT to design multiple comparison testing. We consider absolute and relative predictive ability to test a range of competing risk models, focusing on Value-at-Risk (VaR) and Expected Shortfall (ExS). In devising the test for the absolute predictive ability, we take the route of recent literature and construct balanced simultaneous confidence sets that control for the generalized family-wise error rate, which is the joint probability of rejecting true hypotheses. We implement a step-down method which increases the power of the MHT in isolating false discoveries. In testing for the ExS model predictive ability, we design a new simple test to draw inference about recursive model forecasting capability. In the second suite of statistical testing, we develop a novel device for measuring the relative predictive ability in the bootstrap MHT framework. The device, we coin multiple comparison mapping, provides a statistically robust instrument designed to answer the question: ''which model is the best model?''.

• Item Type: MPRA Paper
• Language: English
• Keywords: value-at-risk, expected shortfall, bootstrap multiple hypothesis testing, generalized familywise error rate, multiple comparison map
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C12 - Hypothesis Testing: General

References

Artzner, P., P.Delbaen, J.M. Eber, and D.Heath, 1999, Coherent measures of risk, Mathematical Finance 9.

Bao, Y., T.Lee, and B.Saltoglu, 2006, Evaluating predictive performance of value-at-risk models in emerging markets: A reality check, Journal of Forecasting 25.

Bao, Y., T.Lee, and B.Saltoglu, 2007, Comparing density forecast models,Journal of Forecasting 26.

Beran, R., 1988, Balanced simultaneous confidence sets, Journal of the American Statistical Association 83.

Beran, R., 1990, Refining bootstrap simultaneous confidence sets, Journal of the American Statistical Association 85.

Beran, R., 2003, The impact of bootstrap on statistical algorithms and theory, Statistical Science 18.

Berkowitz, J., 2001, Testing density forecasts with applications to risk management, Journal of Business and Economic Statistics .

Bierens, H.J., 1987, Kernel estimators of regression functions, in Advances in Econometrics: Fifth World Congress, volume1 (Palgrave, London).

BIS, 2011, Messages from the academic literature on risk measurement for the trading book, Technical Report19, Bank for International Settlements.

Bollerslev, T., 1986, Generalized autoregressive conditional heteroskedasticity, Journal of Econometrics 3.

Christoffersen, P., 1998, Evaluating interval forecasts, International Economic Review 39.

Corradi, V., and N.R. Swanson, 2006, Bootstrap conditional distribution test in the presence of dynamic misspecification, Journal of Econometrics 133.

Corradi, V., and N.R. Swanson, 2007, Nonparametric bootstrap procedures for predictive inference based on recursive estimation schemes, International Economic Review .

Diebold, F.X., and R.S. Mariano, 1995, Comparing predictive accuracy, Journal of Business and Economic Statistics .

Diebold, F.X., A.G. Todd, and A.S. Tay, 1998, Evaluating density forecasts with applications to financial risk management, International Economic Review .

Efron, B., 1979, Bootstrap methods: Another look at the jackknife, The Annals of Statistics 7.

Engle, R., and S.Manganelli, 2004, Caviar: Conditional autoregressive value-at-risk by regression quantile, Journal of Business and Economic Statistics 22.

Engle, R.F., 1982, Autoregressive conditional heteroskedasticity with estimates of the variance of the united kingdom inflation, Econometrica 50.

Glosten, L.R., R.Jagannathan, and D.E. Runkle, 1993, On the relation between the expected value and the volatility of the nominal excess return on stocks, The Journal of Finance 48.

Gonzales-Rivera, G., L.Tae-Hwy, and M.Santosh, 2003, Forecasting volatility: A reality check based on option pricing, utility functions, value-at-risk and predictive likelihood, Working Paper .

Hansen, P.R., 2001, An unbiased and powerful test for superior predictive ability, Working Paper .

Hansen, P.R., and A.Lunde, 2001, A comparison of volatility models: Does anything beat a garch(1,1)?, Working Paper .

Horowitz, J.L., 2000, The bootstrap, Working Paper .

J.P.Morgan, 1996, Riskmetrics$^{TM}$, Technical Report4, J.P.Morgan, Reuters.

Kerkhof, J., and B.Melenberg, 2003, Backtesting for risk-based regulatory capital, Working Paper .

Kerkhof, J., B.Melenberg, and H.Schumacher, 2009, Model risk and capital reserves, Working Paper .

Koenker, B., and G.Bassett, 1978, Regression quantiles, Econometrica 46.

Kupiec, P., 1995, Techniques for verifying the accuracy of risk management models, Journal of Derivatives 3.

Kusch, H.R., 1989, The jackknife and the bootstrap for general stationary observations, The Annals of Statistics 17.

Lahiri, S.N., 1999, Theoretical comparisons of block bootstrap methods, The Annals of Statistics 27.

Lehmann, E.L., and J.Romano, 2005, Generalization of the familiwise error rate, The Annals of Statistics 33.

Liu, R.Y., and K.Singh, 1992, Moving blocks jackknife and bootstrap capture weak dependence, in Exploring the Limits of Bootstrap (John Wiley and Sons).

Longin, F.M., 1996, The asymptotic distribution of extreme stock market returns, The Journal of Business 69.

Mancini, L., and F.Trojani, 2011, Robust value at risk prediction, The Journal of Financial Econometrics 9.

McNeil, A.J., and R.Frey, 2000, Estimation of tail-related risk measures for heteroskedastic financial time series: an extreme value approach, Working Paper .

Nadaraya, E.A., 1964, On estimating regression, Theory of Probability and its Applications 9.

Nelson, D.B., 1991, Conditional heteroskedasticity in asset returns: a new approach, Econometrica 59.

Nordman, D.J., 2009, A note on the stationary bootstrap's variance, The Annals of Statistics 37.

Patton, A., D.N. Politis, and H.White, 2009, Correction to: Automatic block-length selection for dependent bootstrap, Econometric Reviews 28.

Politis, D.N., and J.P. Romano, 1994a, Limit theorems for weakly dependent hilbert space valued random variables with applications to the stationary bootstrap, Statistica Sinica 4.

Politis, D.N., and J.P. Romano, 1994b, The stationary bootstrap, Journal of the American Statistical Association 89.

Politis, D.N., and H.White, 2004, Automatic block-length selection for dependent bootstrap, Econometric Reviews 23.

Romano, J., M.S. Azeem, and M.Wolf, 2009, Hypothesis testing in econometrics, Working Paper Series, Institute for Empirical Research, University of Zurich 444.

Romano, J., and M.Wolf, 2005, Exact and approximate stepdown methods for multiple hypothesis testing, Journal of the American Statistical Association 100.

Romano, J., and M.Wolf, 2007, Control of generalized error rates in multiple testing, The Annals of Statistics 35.

Romano, J., and M.Wolf, 2010, Balanced control of generalized error rates, The Annals of Statistics 38.

Rosenblatt, M., 1952, Remarks on a multivariate transformation, The Annals of Mathmatical Statistics 23.

Vander Vaart, A.W., 1998, Functional delta method, in Asympotic Statistics, chapter20 (Cambridge University Press).

Watson, G.S., 1964, Smooth regression analysis, Sankya: The Indian Journal of Statistics 26.

West, K.D., 1996, Asymptotic inference about predictive ability, Econometrica 64.

White, H., 2000, A reality check for data snooping, Econometrica 68.