Evaluating the Effectiveness of Early Warning Indicators: An Application of Receiver Operating Characteristic Curve Approach to Panel Data

Yildirim, Yusuf and Sanyal, Anirban (2022): Evaluating the Effectiveness of Early Warning Indicators: An Application of Receiver Operating Characteristic Curve Approach to Panel Data.

Abstract

Early warning indicators (EWIs) of banking crises should ideally be judged on how well they function in relation to the choice issue faced by macroprudential policymakers. Several practical features of this challenge are translated into statistical evaluation criteria, including difficulties measuring the costs and advantages of various policy interventions, as well as requirements for the timeliness and stability of EWIs. We analyze the balance panel of possible EWIs for six countries that have experienced currency crisis and banking crisis in recent times. Using possible early warning indicators, we evaluate the suitability of these EWIs in view of their predictive power and stability of signals. The paper observes that credit disbursements to non-financial sectors and central government provides stable signal about systemic risks. Further debt service ratio, inter bank rates and total reserves are also found to be useful in predicting these crisis. Lastly, the paper observes that linear combination of these indicators improves the predictive power of EWIs further.

• Item Type: MPRA Paper
• Language: English
• Keywords: EWIs, ROC, area under the curve, shrinkage
• Subjects: C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics > C40 - General; G - Financial Economics > G0 - General > G01 - Financial Crises; G - Financial Economics > G2 - Financial Institutions and Services > G21 - Banks ; Depository Institutions ; Micro Finance Institutions ; Mortgages

References

[1]Babecky, J., Havranek, T.J., Mateju, J., Rusnak, M., Smidkova, K., Vasicek, B., 2014. Banking, debt, and currency crises in developed countries: stylized facts and early warning indicators. Journal of Financial Stability. 15, 1–17.

[2] Baker, S., Kramer, B., 2007. Peirce, Youden, and Receiver Operating Characteristic Curves. The American Statistician. 61, 343-346. http://dx.doi.org/10.1198/000313007X247643.

[3] Baldwin, R., Giavazzi, F., 2015. The Eurozone crisis. A consensus view of the causes and a few possible solutions, first ed. CEPR Press, London

[4] Basel Committee on Banking Supervision (BCBS), 2010. Guidance for national authorities operating the countercyclical capital buffer.

[5] Behn, M., Detken, C., Peltonen, T. A., Schudel, W., 2013. Setting Countercyclical Capital Buffers Based on Early Warning Models: Would It Work?. ECB Working Paper No. 1604.

[6] Berge, T. J., O, Jorda., 2011. Evaluating the classification of economic activity into recessions and expansions. American Economic Journal: Macroeconomics. 3, 246-277. https://doi.org/10.1257/mac.3.2.246.

[7] Bernanke, B. S., 2004. Gradualism. Remarks at an economics luncheon cosponsored by the Federal Reserve Bank of San Francisco (Seattle Branch) and the University of Washington, Seattle, 20 May 2004.

[8] Borio, C., Drehmann, M., Gambacorta, L., Jim´enez, G., Trucharte, C., 2010. Countercyclical capital buffers: exploring options. BIS Working Paper No. 317.

[9] Chen, S., Svirydzenka, K., 2021. Financial Cycles – Early Warning Indicators of Banking Crises?. IMF Working Papers No. 116.

[10] Cicek, Z., Demirgil, H., 2021. Determination of Poverty Indicators Using Roc Curves in Turkey, Caron, F., Fox, W.P., Chatterjee, A., Hu, N., Klepac, G. in: (Eds.), Analyzing Data Through Probabilistic Modeling in Statistics. IGI Global Inc., Pensilvanya, pp. 1-24.

[11] Cohen, J., Garman, S., Gorr, S., 2009. Empirical calibration of time series monitoring methods using receiver operating characteristic curves. International Journal of Forecasting. 25 (3), 484-497. https://doi.org/10.1016/j.ijforecast.2008.11.007.

[12] Deloitte, 2016. Rand Depreciation [Brochure]. https://www2.deloitte.com/content/dam/Deloitte/za/Documents/process-and-operations/ZA Rand_Depreciation_brochure.pdf (accessed 03.02.2022).

[13] Dembiermont, C., Drehmann, M., Muksakunratana, S., 2013. How much does the private sector really borrow? A new database for total credit to the private non-financial sector. BIS Quarterly Review, 65–81.

[14] Dornbusch, R., 1976. Expectations and Exchange Rate Dynamics. Journal of Political Economy. 84 (6), 1161-1176. http://dx.doi.org/10.1086/260506

[15] Drehmann, M., Borio, C., Tsatsaronis, K., 2011. Anchoring countercyclical capital buffers: The role of credit aggregates. International Journal of Central Banking. 7 (4), 189-240.

[16] Drehmann, M., Juselius, M., 2012. Do debt service costs affect macroeconomic and financial stability?. BIS Quarterly Review, 21-35.

[17] Drehmann, M., Juselius, M., 2014. Evaluating Early Warning Indicators of Banking Crises : Satisfying Policy Requirements. International Journal of Forecasting. 30 (3), 759-780. https://doi.org/10.1016/j.ijforecast.2013.10.002.

[18] Gao, F., Xiong, C., Yan, Y., Yu, K., 2008. Estimating Optimum Linear Combination of Multiple Correlated Diagnostic Tests at a Fixed Specificity with Receiver Operating Characteristic Curves. Journal of Data Science. 6 (1), 1-13. http://dx.doi.org/10.6339/JDS.2008.06(1).368.

[19] Gersl, A., Jasova, M., 2018. Credit-based early warning indicators of banking crises in emerging markets. Economic Systems. 42 (1), 18-31.

[20] Gorr, W., Schneider, M. J., 2011. Large-change forecast accuracy: Reanalysis of M3-competition data using receiver operating characteristic analysis. International Journal of Forecasting. 29 (2), 274–281. http://dx.doi.org/10.1016/j.ijforecast.2012.12.001.

[21] Gourinchas, P. O., Obstfeld, M., 2012. Stories of the twentieth century for the twenty-first. American Economic Journal: Macroeconomics. 4 (1), 226-265. https://doi.org/10.1257/mac.4.1.226.

[22] Jorda, O., Schularick, M., Taylor, A. M., 2011. Financial crises, credit booms, and external imbalances: 140 years of lessons. IMF Economic Review. 59 (2), 340-378. https://www.jstor.org/stable/41290964.

[23] Hahm, J., Shin, H. S., Shin, K., 2012. Non-core Bank Liabilities and Financial Vulnerability. Journal of Money, Credit and Banking. 45 (1), 3-36. https://doi.org/10.1111/jmcb.12035.

[24] Harvard Business School (HBS), 2022. Global Crises Data by Country. [Data set]. https://www.hbs.edu/behavioral-finance-and-financial-stability/data/Pages/global.aspx.

[25] Liu, C., Liu, A., Halabi, S., 2011. A min-max combination of biomarkers to improve diagnostic accuracy. Statistics in Medicine. 30 (16), 2005 - 2014. http://dx.doi.org/10.1002/sim.4238.

[26] Nelson, R. M., 2018. Turkey’s Currency Crisis. Congressional Research Service. URL https://sgp.fas.org/crs/mideast/IF10957.pdf (accessed 3.2.22).

[27] Orhangazi, O., 2014. Capital Flows and Credit Expansions in Turkey. Review of Radical Political Economics. 46 (4), 509 – 516. https://doi.org/10.1177/0486613414537993.

[28] Pepe, M.S., Thompson, M.L., 2000. Combining diagnostic test results to increase accuracy. Biostatistics. 1 (2), 123–140. http://dx.doi.org/10.1093/biostatistics/1.2.123.

[29] Reinhart, C. M., Rogoff, K. S., 2009. This time is different: Eight centuries of financial folly, first ed. Princeton University Press, Princeton and Oxford.

[30] Romano, S., 2021. The 2011 Crisis in Italy: A Story of Deep-Rooted (and Still Unresolved) Economic and Political Weaknesses, in: De Souza Guilherme, B., Ghymers, C., Griffith-Jones S., Ribeiro Hoffmann A. (Eds.), Financial Crisis Management and Democracy. Springer, Cham, pp. 173-184 https://doi.org/10.1007/978-3-030-54895-7 10.

[31] Reiser, B., Faraggi, D., 1997. Confidence intervals for the generalized ROC criterion. Biometrics. 53 (2), 644–652. http://dx.doi.org/10.2307/2533964.

[32] Sameera, G., Vardhan, R.V., Sarma, K.V., 2016. Binary classification using multivariate receiver operating characteristic curve for continuous data. Journal of Biopharmaceutical Statistics. 26 (3), 421 – 431. http://dx.doi.org/10.1080/10543406.2015.1052479.

[33] Su, J. Q., Liu, S., 1993. Linear combinations of multiple diagnostic markers. Journal of the American Statistical Association. 88 (424), 1350-1355. http://dx.doi.org/10.1080/01621459.1993.10476417.

[34] Vartanian, P. R., Garbe, H., 2019. The Brazilian Economic Crisis during the Period 2014-2016: Is There Precedence of Internal or External Factors?. Journal of International and Global Economic Studies. 12 (1), 66-86.

[35] Viktorov, I., Abramov, A., 2020. The 2014–15 Financial Crisis in Russia and the Foundations of Weak Monetary Power Autonomy in the International Political Economy. New Political Economy. 25 (4), 487–510. https://doi.org/10.1080/13563467.2019.1613349

[36] Witte, D. M., 2012. What Effect Did the Credit Crisis of 2008 Have on European Exchange Rates?. Eastern European Economics. 50 (3), 79-93. https://doi.org/10.2753/eee0012-8775500304.

[37] Yasar, H., Kilimci, Z.H., 2020. US Dollar/Turkish Lira Exchange Rate Forecasting Model Based on Deep Learning Methodologies and Time Series Analysis. Symmetry. 12 (9), 1 – 18. http://dx.doi.org/10.3390/sym12091553.

[38] Yu, W., Park, T., 2015. Two simple algorithms on linear combination of multiple biomarkers to maximize partial area under the ROC curve. Computational Statistics Data Analysis. 88, 15-27. http://dx.doi.org/10.1016/j.csda.2014.12.002.