The predictive accuracy of credit ratings: Measurement and statistical inference

Orth, Walter (2010): The predictive accuracy of credit ratings: Measurement and statistical inference.

Abstract

Credit ratings are ordinal predictions for the default risk of an obligor. To evaluate the accuracy of such predictions commonly used measures are the Accuracy Ratio or, equivalently, the Area under the ROC curve. The disadvantage of these measures is that they treat default as a binary variable thereby neglecting the timing of the default events and also not using the full information from censored observations. We present an alternative measure that is related to the Accuracy Ratio but does not suffer from these drawbacks. As a second contribution, we study statistical inference for the Accuracy Ratio and the proposed measure in the case of multiple cohorts of obligors with overlapping lifetimes. We derive methods that use more sample information and lead to more powerful tests than alternatives that filter just the independent part of the dataset. All procedures are illustrated in the empirical section using a dataset of S\&P Long Term Credit Ratings.

• Item Type: MPRA Paper
• Language: English
• Keywords: Ratings; predictive accuracy; Accuracy Ratio; Harrell's C; overlapping lifetimes
• Subjects: C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics > C41 - Duration Analysis ; Optimal Timing Strategies; G - Financial Economics > G3 - Corporate Finance and Governance > G32 - Financing Policy ; Financial Risk and Risk Management ; Capital and Ownership Structure ; Value of Firms ; Goodwill; G - Financial Economics > G2 - Financial Institutions and Services > G24 - Investment Banking ; Venture Capital ; Brokerage ; Ratings and Ratings Agencies

References

Arvesen, J. N. (1969). Jackknifing U-statistics. Annals of Mathematical Statistics, 40 (6), 2076–2100.

Bamber, D. (1975). The Area above the Ordinal Dominance Graph and the Area Below the Receiver Operating Characteristic Graph. Journal of Mathematical Psychology, 12, 387–415.

Banasik, J., Crook, J. N., & Thomas, L. C. (1999). Not if but When Borrowers Default. Journal of the Operational Research Society, 50, 12, 1185–1190.

Basel Committee on Banking Supervision (2006). International Convergence of Capital Measurement and Capital Standards – A Revised Framework.

Busing, F. M. T. A., Meijer, E., & van der Leeden, R. (1999). Delete-m Jackknife for Unequal m. Statistics and Computing, 9, 3–8.

Cantor, R., Hamilton, D. T., & Tennant, J. (2008). Confidence Intervals for Corporate Default Rates. Risk, March 2008, 93–98.

Cantor, R. & Mann, C. (2003). Measuring the Performance of Corporate Bond Ratings. Special comment, Moody’s Investors Service.

DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics, 44(3), 837–845.

Engelmann, B., Hayden, E., & Tasche, D. (2003). Testing Rating Accuracy. Risk, January 2003, 82–86.

Field, C. A. & Welsh, A. H. (2007). Bootstrapping Clustered Data. Journal of the Royal Statistical Society Series B, 69, 369–390.

Guettler, A. & Raupach, P. (2010). The Impact of Downward Rating Momentum. Journal of Financial Services Research, 37, 1–23.

Harrell, F. E. J., Lee, K. L., & Mark, D. B. (1996). Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors. Statistics in Medicine, 15, 361–387.

Hoeffding, W. (1948). A Class of Statistics with Asymptotically Normal Distribution. Annals of Mathematical Statistics, 19, 293–325.

Horowitz, J. L. (2001). The Bootstrap. Handbook of Econometrics, 5, 3159–3228.

Kraemer, W. & Guettler, A. (2008). On Comparing the Accuracy of Default Predictions in the Rating Industry. Empirical Economics, 34, 343–356.

Lando, D. & Skodeberg, T. M. (2002). Analyzing Rating Transitions and Rating Drift with Continous Observations. Journal of Banking & Finance, 26, 423–444.

Newson, R. (2006). Confidence Intervals for Rank Statistics: Somers’ D and Extensions. The Stata Journal, 6(3), 309–334.

Pencina, M. J. & D’Agostino, R. B. (2004). Overall C as a Measure of Discrimination in Survival Analysis: Model Specific Population Value and Confidence Intervals. Statistics in Medicine, 23, 2109–2123.

Shao, J. & Tu, D. (1996). The Jackknife and Bootstrap. Springer.

Shumway, R. H. & Stoffer, D. S. (2006). Time Series Analysis and Its Applications. Springer, 2nd edition.

Somers, R. H. (1962). A New Asymmetric Measure of Association for Ordinal Variables. American Sociological Review, 27(6), 799–811.

Standard & Poor’s (2010). The Time Dimension of Standard & Poor’s Credit Ratings. Global Credit Portal, September 22, 2010.

Thomas, L. C., Edelman, D. B., & Crook, J. B. (2002). Credit Scoring and its Applications. SIAM.