Monotonic Estimation for Probability Distribution and Multivariate Risk Scales by Constrained Minimum Generalized Cross-Entropy

Yang, Bill Huajian (2019): Monotonic Estimation for Probability Distribution and Multivariate Risk Scales by Constrained Minimum Generalized Cross-Entropy. Forthcoming in: International Journal of Machine Learning and Computing

Abstract

Minimum cross-entropy estimation is an extension to the maximum likelihood estimation for multinomial probabilities. Given a probability distribution {r_i }_(i=1)^k, we show in this paper that the monotonic estimates {p_i }_(i=1)^k for the probability distribution by minimum cross-entropy are each given by the simple average of the given distribution values over some consecutive indexes. Results extend to the monotonic estimation for multivariate outcomes by generalized cross-entropy. These estimates are the exact solution for the corresponding constrained optimization and coincide with the monotonic estimates by least squares. A non-parametric algorithm for the exact solution is proposed. The algorithm is compared to the “pool adjacent violators” algorithm in least squares case for the isotonic regression problem. Applications to monotonic estimation of migration matrices and risk scales for multivariate outcomes are discussed.

• Item Type: MPRA Paper
• Language: English
• Keywords: maximum likelihood, cross-entropy, least squares, isotonic regression, constrained optimization, multivariate risk scales
• Subjects: C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C13 - Estimation: General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C18 - Methodological Issues: General; C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics; C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics > C44 - Operations Research ; Statistical Decision Theory; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C51 - Model Construction and Estimation; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C52 - Model Evaluation, Validation, and Selection; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C53 - Forecasting and Prediction Methods ; Simulation Methods; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C54 - Quantitative Policy Modeling; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C55 - Large Data Sets: Modeling and Analysis; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C58 - Financial Econometrics; C - Mathematical and Quantitative Methods > C6 - Mathematical Methods ; Programming Models ; Mathematical and Simulation Modeling > C61 - Optimization Techniques ; Programming Models ; Dynamic Analysis; C - Mathematical and Quantitative Methods > C6 - Mathematical Methods ; Programming Models ; Mathematical and Simulation Modeling > C63 - Computational Techniques ; Simulation Modeling

References

[1] Barlow, R. E.; Bartholomew, D. J.; Bremner, J. M.; Brunk, H. D. (1972). Statistical inference under order restrictions; the theory and application of isotonic regression. New York: Wiley. ISBN 0-471-04970-0.

[2] Best, M.J.; Chakravarti N. (1990). Active set algorithms for isotonic regression; a unifying framework. Mathematical Programming. 47: 425–439. doi:10.1007/BF0158087

[3] Friedman, J. and Tibshirani, R. (1984). The Monotone Smoothing of Scatterplots. Technometrics, Vol. 26, No. 3, pp. 243-250. DOI: 10.2307/1267550

[4] Leeuw, J. D, Hornik, K. and Mair, P. (2009). Isotone Optimization in R: Pool-Adjacent- Violators Algorithm (PAVA) and Active Set Methods, Journal of statistical software 32(5). DOI: 10.18637/jssw.v032.i05

[5] Yang, B. H. (2018). Smoothing Algorithms by Constrained Maximum Likelihood, Journal of Risk Model Validation, Volume 12 (2), pp. 89-102.

[6] Potharst, R. and Feelders, A. J. (2002). Classification Trees for Problems with Monotonicity Constraints, SIGKDD Explorations, Vol. 14 (1): 1-10, 2002

[7] Kotlowski, W. and Slowinski, R. (2009). Rule Learning with Monotonicity Constraints, Proceedings of the 26th Annual International Conference on Machine Learning, pp. 537-544, 2009

[8] Eichenberg, T. (2018). Supervised Weight of Evidence Binning of Numeric Variables and Factors, R-Package Woebinning.

[9] You, S.; Ding, D.; Canini, K.; Pfeifer, J. and Gupta, M. (2017). Deep Lattice Networks and Partial Monotonic Functions, 31st Conference on Neural Information Processing System (NIPS), 2017

[10] Kullback, S.; Leibler, R.A. (1951). On information and sufficiency. Annals of Mathematical Statistics. 22 (1): 79–86. doi:10.1214/aoms/1177729694.

[11] Goodfellow, I..; Bengio, Y.; Courville, A. (2016). Deep Learning. MIT Press.

[12] Murphy, K. (2012). Machine Learning: A Probabilistic Perspective. MIT. ISBN 978-0262018029.

[13] SAS Institute Inc (2014). SAS/STAT(R) 13.2 User’s Guide

[14] Robertson, T.; Wright, F. T.; Dykstra, R. L. (1998). Order Restricted Statistical Inference, John Wiley & Son.

[15] Nocedal, J., and Wright, S. J. (2006). Numerical Optimization, 2nd edn. Springer

[16] Nelson, C. R, Siegel, A. F. (1987), Parsimonious Modeling of Yield Curves, Journal of Business, Vol. 60, N0. 4, 473-489