Marginal Likelihood Estimation with the Cross-Entropy Method

Chan, Joshua and Eisenstat, Eric (2012): Marginal Likelihood Estimation with the Cross-Entropy Method.

Abstract

We consider an adaptive importance sampling approach to estimating the marginal likelihood, a quantity that is fundamental in Bayesian model comparison and Bayesian model averaging. This approach is motivated by the difficulty of obtaining an accurate estimate through existing algorithms that use Markov chain Monte Carlo (MCMC) draws, where the draws are typically costly to obtain and highly correlated in high-dimensional settings. In contrast, we use the cross-entropy (CE) method, a versatile adaptive Monte Carlo algorithm originally developed for rare-event simulation. The main advantage of the importance sampling approach is that random samples can be obtained from some convenient density with little additional costs. As we are generating independent draws instead of correlated MCMC draws, the increase in simulation effort is much smaller should one wish to reduce the numerical standard error of the estimator. Moreover, the importance density derived via the CE method is in a well-defined sense optimal. We demonstrate the utility of the proposed approach by two empirical applications involving women's labor market participation and U.S. macroeconomic time series. In both applications the proposed CE method compares favorably to existing estimators.

• Item Type: MPRA Paper
• Language: English
• Keywords: importance sampling, model selection, probit, logit, time-varying parameter vector autoregressive model, dynamic factor model
• Subjects: C - Mathematical and Quantitative Methods > C3 - Multiple or Simultaneous Equation Models ; Multiple Variables > C32 - Time-Series Models ; Dynamic Quantile Regressions ; Dynamic Treatment Effect Models ; Diffusion Processes ; State Space Models; C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C52 - Model Evaluation, Validation, and Selection; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C15 - Statistical Simulation Methods: General; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C11 - Bayesian Analysis: General

References

D. Ardia, N. Basturk, L. Hoogerheide, and H. K. van Dijk. A comparative study of Monte Carlo methods for efficient evaluation of marginal likelihood. Computational Statistics and Data Analysis, 2010. In press.

K. J. Arrow. Essays in the Theory of Risk Bearing. North-Holland, Amsterdam, 1970.

S. Asmussen, R. Y. Rubinstein, and D. P. Kroese. Heavy tails, importance sampling and cross-entropy. Stochastic Models, 21:57–76, 2005.

B. Bernanke, J. Boivin, and P. S. Eliasz. Measuring the effects of monetary policy: A factor-augmented vector autoregressive (FAVAR) approach. The Quarterly Journal of Economics, 120(1):387–422, 2005.

Z. I. Botev and D. P. Kroese. The generalized cross-entropy method, with applications to probability density estimation. Methodology and Computing in Applied Probability, 13:1–27, 2011.

F. Canova. Modelling and forecasting exchange rates with a Bayesian time-varying coefficient model. Journal of Economic Dynamics and Control, 17:233–262, 1992.

C. K. Carter and R. Kohn. On Gibbs sampling for state space models. Biometrika, 81:541–553, 1994.

J. C. C. Chan and I. Jeliazkov. Efficient simulation and integrated likelihood estimation in state space models. International Journal of Mathematical Modelling and Numerical Optimisation, 1:101–120, 2009.

J. C. C. Chan and D. P. Kroese. Efficient estimation of large portfolio loss probabilities in t-copula models. European Journal of Operational Research, 205:361–367, 2010.

J. C. C. Chan, P. W. Glynn, and D. P. Kroese. A comparison of cross-entropy and variance minimization strategies. Journal of Applied Probability, 48A:183–194, 2011.

S. Chib. Marginal likelihood from the Gibbs output. Journal of the American Statistical Association, 90:1313–1321, 1995.

S. Chib and I. Jeliazkov. Marginal likelihood from the Metropolis-Hastings output. Journal of the American Statistical Association, 96:270–281, 2001.

P. T. de Boer, D. P. Kroese, and R. Y. Rubinstein. A fast cross-entropy method for estimating buffer overflows in queueing networks. Management Science, 50:883–895, 2004.

N. Friel and A. N. Pettitt. Marginal likelihood estimation via power posteriors. Journal Royal Statistical Society Series B, 70:589–607, 2008.

S. Fruhwirth-Schnatter and Helga Wagner. Marginal likelihoods for non-Gaussian models using auxiliary mixture sampling. Computational Statistics and Data Analysis, 52(10):4608 – 4624, 2008.

A. E. Gelfand and D. K. Dey. Bayesian model choice: Asymptotics and exact calculations. Journal of the Royal Statistical Society Series B, 56(3):501–514, 1994.

A. Gelman and X. Meng. Simulating normalizing constants: From importance sampling to bridge sampling to path sampling. Statistical Science, 13:163–185, 1998.

A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin. Bayesian Data Analysis. Chapman & Hall, New York, second edition, 2003.

J. Geweke. Exact predictive densities for linear models with ARCH disturbances. Journal of Econometrics, 40(1):63 – 86, 1989.

J. Geweke. Using simulation methods for Bayesian econometric models: inference, development, and communication. Econometric Reviews, 18(1):1–73, 1999.

A. Golan. Information and entropy econometrics – a review and synthesis. Foundations and Trends in Econometrics, 2(1-2):1–145, 2008.

A. Golan, G. G. Judge, and D. Miller. Maximum Entropy Econometrics: Robust Estimation with Limited Data. Wiley, New York, 1996.

C. Han and B. P. Carlin. Markov chain Monte Carlo methods for computing Bayes factors: a comparative review. Journal of the American Statistical Association, 96:1122–1132, 2001.

L. F. Hoogerheide, J. F. Kaashoek, and H. K. van Dijk. On the shape of posterior densities and credible sets in instrumental variable regression models with reduced rank: An application of flexible sampling methods using neural networks. Journal of Econometrics, 139(1):154–180, 2007.

K-P. Hui, N. Bean, M. Kraetzl, and D. P. Kroese. The cross-entropy method for network reliability estimation. Annals of Operations Research, 134:101–118, 2005.

I. Jeliazkov and R. Liu. A model-based ranking of U.S. recessions. Economics Bulletin, 30(3): 2289–2296, 2010.

J. M. Keith, D. P. Kroese, and G. Y. Sofronov. Adaptive independence samplers. Statistics and Computing, 18:409–420, 2008.

S. Kim, N. Shephard, and S. Chib. Stochastic volatility: Likelihood inference and comparison with ARCH models. Review of Economic Studies, 65(3):361–393, 1998.

G. Koop. Bayesian Econometrics. Wiley & Sons, New York, 2003.

G. Koop and D. Korobilis. Bayesian multivariate time series methods for empirical macroeconomics. Foundations and Trends in Econometrics, 3(4):267–358, 2010.

D. P. Kroese. The cross-entropy method. In James J. Cochran, Louis A. Cox, Pinar Keskinocak, Jeffrey P. Kharoufeh, and J. Cole Smith, editors, Wiley Encyclopedia of Operations Research and Management Science. Wiley & Sons, New York, 2010.

D. P. Kroese and R. Y. Rubinstein. The transform likelihood ratio method for rare event simulation with heavy tails. Queueing Systems, 46:317–351, 2004.

D. P. Kroese, T. Taimre, and Z. I. Botev. Handbook of Monte Carlo Methods. John Wiley & Sons, New York, 2011.

E. Maasoumi. A compendium to information theory in economics and econometrics. Econometric Reviews, 12:137–181, 1993.

J.Marschak. Economics of information systems. Journal of the American Statistical Association, 66:192–219, 1971.

T. A. Mroz. The sensitivity of an empirical model of married women’s hours of work to economic and statistical assumptions. Econometrica, 55:765–799, 1987.

M. A. Newton and A. E. Raftery. Approximate Bayesian inference with the weighted likelihood bootstrap. Journal of the Royal Statistical Society Series B, 56:3–48, 1994.

R. Y. Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operational Research, 99:89–112, 1997.

R. Y. Rubinstein and D. P. Kroese. The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization Monte-Carlo Simulation, and Machine Learning. Springer-Verlag, New York, 2004.

C. A. Sims. Macroeconomics and reality. Econometrica, 48:1–48, 1980.

A. Zellner. Bayesian methods and entropy in economics and econometrics. In W. T. Grandy and L. H. Schick, editors, Maximum Entropy and Bayesian Methods, volume 43 of Fundamental Theories of Physics, pages 17–31. Springer Netherlands, 1991.