A Proposal for a Unified Forecast Accuracy Index (UFAI): Toward Multidimensional and Context-Aware Forecast Evaluation

Chellai, Fatih (2025): A Proposal for a Unified Forecast Accuracy Index (UFAI): Toward Multidimensional and Context-Aware Forecast Evaluation.

Abstract

Forecast accuracy evaluation is a cornerstone in fields as diverse as finance, public health, energy, and meteorology. However, traditional reliance on single-error metrics—such as MAE, RMSE, or MAPE—offers only a fragmented view of a model’s performance, often obscuring critical dimensions like systematic bias, volatility, directional behavior, or shape fidelity. To overcome these limitations, this study proposes the Unified Forecast Accuracy Index (UFAI), a multidimensional and composite metric that consolidates several facets of forecasting quality into a single, interpretable score. UFAI integrates four normalized sub-indices—bias, variance, directional accuracy, and shape preservation—each capturing a distinct performance characteristic. The framework accommodates multiple weighting schemes: equal weighting for simplicity, expert-informed weighting to reflect domain-specific priorities, and data-driven weighting based on statistical principles such as Principal Component Analysis and entropy measures. This flexibility enables users to adapt the index to diverse forecasting objectives and application contexts. The article details the mathematical formulation of each sub-index, discusses the theoretical soundness and practical implications of different weighting strategies, and demonstrates the utility of UFAI through comparative model evaluations. Emphasis is placed on the index’s normalization, interpretability, robustness to outliers, and extensibility to future use cases such as multi-horizon and probabilistic forecasts. By offering a more integrated and context-aware assessment tool, the UFAI marks a significant advancement in forecast evaluation methodology, supporting more reliable model selection and ultimately enhancing decision-making in data-driven environments.

• Item Type: MPRA Paper
• Language: English
• Keywords: Forecast evaluation, Unified forecast accuracy index, Composite metrics, Model comparison
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General; C - Mathematical and Quantitative Methods > C2 - Single Equation Models ; Single Variables; C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics

References

(1) Armstrong, J. S. (2001). Principles of Forecasting. Springer. (2) Armstrong, J. S., & Collopy, F. (1992). Error measures for generalizing about forecasting methods: Empirical comparisons. International Journal of Forecasting, 8(1), 69–80. https://doi.org/10.1016/0169-2070(92)90052-W (3) Belton, V., & Stewart, T. J. (2002). Multiple Criteria Decision Analysis: An Integrated Approach. Springer. (4) Bontempi, G., Taieb, S. B., & Le Borgne, Y.-A. (2012). Machine learning strategies for time series forecasting. European Business Intelligence Summer School. (5) Chicco, D., & Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21(6). https://doi.org/10.1186/s12864-019-6413-7 (6) De Myttenaere, A., Golden, B., Le Grand, B., & Rossi, F. (2016). Mean absolute percentage error for regression models. Neurocomputing, 192, 38–48. https://doi.org/10.1016/j.neucom.2015.12.114 (7) Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. https://doi.org/10.1198/016214506000001437 (8) Gokgoz, F., & Ozgür, L. (2020). An integrated MCDA framework for forecasting model selection. Expert Systems with Applications, 145, 113115. https://doi.org/10.1016/j.eswa.2020.113115 (9) Hong, T., Pinson, P., & Fan, S. (2016). Global energy forecasting competition 2012. International Journal of Forecasting, 32(3), 896–913. https://doi.org/10.1016/j.ijforecast.2016.04.007 (10) Hyndman, R. J., & Koehler, A. B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22(4), 679–688. https://doi.org/10.1016/j.ijforecast.2006.03.001 (11) Jolliffe, I. T. (2002). Principal Component Analysis (2nd ed.). Springer. (12) Keeney, R. L., & Raiffa, H. (1976). Decisions with Multiple Objectives: Preferences and Value Trade-Offs. Wiley. (13) Makridakis, S. (1993). Accuracy measures: Theoretical and practical concerns. International Journal of Forecasting, 9(4), 527–529. https://doi.org/10.1016/0169-2070(93)90009-L (14) Makridakis, S., & Hibon, M. (2000). The M3-Competition: Results, conclusions and implications. International Journal of Forecasting, 16(4), 451–476. https://doi.org/10.1016/S0169-2070(00)00074-0 (15) Makridakis, S., Spiliotis, E., & Assimakopoulos, V. (2018). Statistical and Machine Learning forecasting methods: Concerns and ways forward. PLOS ONE, 13(3), e0194889. https://doi.org/10.1371/journal.pone.0194889 (16) OECD. (2008). Handbook on Constructing Composite Indicators. OECD Publishing. (17) Pesaran, M. H., & Timmermann, A. (1992). A simple nonparametric test of predictive performance. Journal of Business & Economic Statistics, 10(4), 461–465. https://doi.org/10.2307/1391710 (18) Powers, D. M. W. (2011). Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation. Journal of Machine Learning Technologies, 2(1), 37–63. (19) Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379–423. (20) Stock, J. H., & Watson, M. W. (2001). Vector autoregressions. Journal of Economic Perspectives, 15(4), 101–115. https://doi.org/10.1257/jep.15.4.101 (21) Theil, H. (1966). Applied Economic Forecasting. North-Holland. (22) UNDP. (1990). Human Development Report. (23) Yale Center for Environmental Law & Policy. (2016). 2016 Environmental Performance Index. (24) Zavadskas, E. K., & Turskis, Z. (2011). Multiple criteria decision making (MCDM) methods in economics: An overview. Technological and Economic Development of Economy, 17(2), 397–427. https://doi.org/10.3846/13928619.2011.586622