Model Evaluation for Policy Insights: Reflections on the Forum Process

Huntington, Hillard G. (2021): Model Evaluation for Policy Insights: Reflections on the Forum Process. Forthcoming in: Energy Policy , Vol. 156, (September 2021): p. 112365.

Abstract

Model evaluation is best considered as a process for communicating with the policymaking or policy-advising community. Six decades of energy modelling have witnessed increasing complexity in these systems, a situation that raises a number of important challenges in using them effectively in policymaking organizations. When used as a learning rather than forecasting tool, these systems can be evaluated individually one by one or through joint efforts to compare them in multi-model exercises. After summarizing the evolution of energy modelling and efforts to evaluate them since the first oil embargo, this essay provides a guide to future evaluation collaborations by highlighting a few challenges that would improve the value of these studies for the policymaking community. These challenges range broadly and cover topics such as enhancing the engagement of the model user, ventilating the models’ complexity with intuitive insights, using simple models to demonstrate key parameters or responses, applying judicious occasional meta-analysis when there is value added, reporting model responses and calibrating them for decisionmakers, considering retrospective evaluation for a past period (when possible), selecting standardized or modeler-choice baseline conditions, selectively developing policy or diagnostic alternative cases, and institutionalising the model evaluation process for a specific topic or region.

• Item Type: MPRA Paper
• Language: English
• Keywords: Energy Policymaking; Model Evaluation; Model Comparison
• Subjects: C - Mathematical and Quantitative Methods > C5 - Econometric Modeling > C54 - Quantitative Policy Modeling; Q - Agricultural and Natural Resource Economics ; Environmental and Ecological Economics > Q4 - Energy > Q41 - Demand and Supply ; Prices; Q - Agricultural and Natural Resource Economics ; Environmental and Ecological Economics > Q5 - Environmental Economics > Q54 - Climate ; Natural Disasters and Their Management ; Global Warming

References

AI Impacts, n.d. Trends in the cost of computing [WWW Document]. URL https://aiimpacts.org/trends-in-the-cost-of-computing/ (accessed 12.6.20).

Bistline, J.E., Hodson, E., Rossmann, C.G., Creason, J., Murray, B., Barron, A.R., 2018. Electric sector policy, technological change, and U.S. emissions reductions goals: Results from the EMF 32 model intercomparison project. Energy Econ. 73, 307–325. https://doi.org/10.1016/J.ENECO.2018.04.012

Bistline, J.E.T., Blanford, G.J., 2020. Value of technology in the U.S. electric power sector: Impacts of full portfolios and technological change on the costs of meeting decarbonization goals. Energy Econ. 86. https://doi.org/10.1016/j.eneco.2020.104694

Cropper, M.L., Morgenstern, R.D., Rivers, N., 2018. Facilitating retrospective analysis of environmental regulations. Rev. Environ. Econ. Policy 12, 359–370. https://doi.org/10.1093/reep/rey011

DeCarolis, J., Daly, H., Dodds, P., Keppo, I., Li, F., McDowall, W., Pye, S., Strachan, N., Trutnevyte, E., Usher, W., Winning, M., Yeh, S., Zeyringer, M., 2017. Formalizing best practice for energy system optimization modelling. Appl. Energy 194, 184–198. https://doi.org/10.1016/j.apenergy.2017.03.001

DeCarolis, J.F., Hunter, K., Sreepathi, S., 2012. The case for repeatable analysis with energy economy optimization models. Energy Econ. 34, 1845–1853. https://doi.org/10.1016/j.eneco.2012.07.004

Edelenbosch, O.Y., van Vuuren, D.P., Bertram, C., Carrara, S., Emmerling, J., Daly, H., Kitous, A., McCollum, D.L., Saadi Failali, N., 2017. Transport fuel demand responses to fuel price and income projections: Comparison of integrated assessment models. Transp. Res. Part D Transp. Environ. 55, 310–321. https://doi.org/10.1016/j.trd.2017.03.005

Edmonds, J., Reilly, J., 1983. A long-term global energy- economic model of carbon dioxide release from fossil fuel use. Energy Econ. 5, 74–88. https://doi.org/10.1016/0140-9883(83)90014-2

Energy Modeling Forum, 2015. North American Natural Gas Markets in Transition. Stanford University, Stanford, CA.

Energy Modeling Forum, 2013. Changing the Game? Emissions and Market Implications of New Natural Gas Supplies. Stanford University, Stanford, CA.

Energy Modeling Forum, 1991. International Oil Supplies and Demands. Stanford University, Stanford, CA.

Energy Modeling Forum, 1981. Aggregate elasticity of energy demand. Energy J. 2, 37–75. https://doi.org/10.5547/issn0195-6574-ej-vol2-no2-3

Energy Modeling Forum, 1977. Energy and the Economy. Stanford University, Stanford, CA.

Fawcett, A.A., Clarke, L.E., Rausch, S., Weyant, J.P., 2014. Overview of EMF 24 Policy Scenarios. Energy J. 35, 33–60. https://doi.org/10.5547/01956574.35.SI1.3

Fischer, C., Morgenstern, R.D., 2006. Carbon abatement costs: Why the wide range of estimates? Energy J. 27, 73–86. https://doi.org/10.5547/ISSN0195-6574-EJ-Vol27-No2-5

Frankel, J. (2008), Serious research balances economic costs & environmental benefits of climate policy, Views on the Economy and the World (blog), Aug. https://www.belfercenter.org/publication/serious-research-balances-economic-costs-environmental-benefits-climate-policy

Fromm, G., Klein, L.R., 1973. A comparision of eleven econometric models of the United States. Am. Econ. Rev. 63 (2), 385–393.

Greenberger, M., Brewer, G.D., Hogan, W.W., Russell, M., 1983. Caught Unawares: The Energy Decade in Retrospect. Ballinger, Cambridge,MA.

Greenberger, M., Richels, R., 1979. Assessing energy policy models: current state and future directions. Annu. Rev. Energy 4, 467–500. https://doi.org/10.1146/annurev.eg.04.110179.002343

Groissböck, M., 2019. Are open source energy system optimization tools mature enough for serious use? Renew. Sustain. Energy Rev. 102, 234–248. https://doi.org/10.1016/J.RSER.2018.11.020

Hoffman, K.C., Jorgenson, D.W., 1977. Economic and technological models for evaluation of energy policy. Bell J. Econ. 8, 444–466. https://doi.org/10.2307/3003296

Hogan, W.W., 2002. Energy modeling for policy studies. Oper. Res. 50, 89–95. https://doi.org/10.1287/opre.50.1.89.17803

Howells, M., Rogner, H., Strachan, N., Heaps, C., Huntington, H., Kypreos, S., Hughes, A., Silveira, S., DeCarolis, J., Bazillian, M., Roehrl, A., 2011. OSeMOSYS: The Open Source Energy Modeling System: An introduction to its ethos, structure and development. Energy Policy 39, 5850–5870. https://doi.org/10.1016/j.enpol.2011.06.033

Huntington, H.G., 1994. Oil price forecasting in the 1980s: What went wrong. Energy J. 15, 1–22.

Huntington, H.G., Weyant, J.P., Sweeney, J.L., 1982. Modeling for insights, not numbers: the experiences of the energy modeling forum. Omega 10, 449–462.

Integrated Assessment Modeling Consortium, n.d. IAMC [WWW Document]. URL https://www.iamconsortium.org/ (accessed 12.6.20).

IPCC, 2014. Climate Change Mitigation. Fifth Assessment Report (AR5), Working Group III, Intergovernmental Panel on Climate Change.

IWG (Interagency Working Group). 2010. Interagency Working Group on Social Cost of Carbon, US Government, technical support document: Social cost of carbon for regulatory impact analysis under Executive Order 12866. http://www.epa.gov/oms/climate/regulations/scc-tsd.pdf

IWG (Interagency Working Group). 2016. Technical Update of the Social Cost of Carbon for Regulatory Impact Analysis, US Government, technical support document under Executive Order 12866. https://www.epa.gov/sites/production/files/2016-12/documents/sc_co2_tsd_august_2016.pdf

Kay, J., King, M., 2020. Radical uncertainty: decision-making beyond the numbers, New York: W.W. Norton and Company, 544pp.

Lempert, R.J., 2019. Robust decision making (RDM), in: Marchau, V., Walker, W., Bloemen, P., Popper, S. (Eds.), Decision Making under Deep Uncertainty. Springer, Cham., pp. 23–51. https://doi.org/10.1007/978-3-030-05252-2_2

Mai, T., Bistline, J., Sun, Y., Cole, W., Marcy, C., Namovicz, C., Young, D., 2018. The role of input assumptions and model structures in projections of variable renewable energy: A multi-model perspective of the U.S. electricity system. Energy Econ. 76, 313–324. https://doi.org/10.1016/j.eneco.2018.10.019

Manne, A.S., 1979. ETA-MACRO: A model of energy-economy interactions, in: ACS Symposium Series. JAI Press, pp. 205–233.

Manne, A.S., 1976. ETA: A model for energy technology assessment. Bell J Econ 7, 379–406. https://doi.org/10.2307/3003263

Manne, A.S., Richels, R.G., 1991. Buying greenhouse insurance. Energy Policy 19, 543–552. https://doi.org/10.1016/0301-4215(91)90034-L

Massachusetts Institute of Technology, 1978. Independent assessment of energy policy models : two case studies. Cambridge, MA.

Metcalf, G.E., Stock, J.H., 2017. Integrated assessment models and the social cost of carbon: A review and assessment of U.S. experience. Rev. Environ. Econ. Policy 11(1):80–99. https://doi.org/10.1093/reep/rew014

Naill, R.F., 1992. A system dynamics model for national energy policy planning. Syst. Dyn. Rev. 8, 1–19. https://doi.org/10.1002/sdr.4260080102

National Research Council, 1978. Energy Modeling for an Uncertain Future. National Academies Press, Washington, DC. https://doi.org/10.17226/19960

Nordhaus, W., 2014. Estimates of the Social Cost of Carbon: Concepts and Results from the DICE-2013R Model and Alternative Approaches. J. Assoc. Environ. Resour. Econ. 1(1/2):273–312. https://doi.org/10.1086/676035

Nordhaus, W., 1993. Optimal greenhouse-gas reductions and tax policy in the “Dice” model. Am. Econ. Rev. 83(2), 313–317.

Nordhaus, W.D., 2007. Two centuries of productivity growth in computing. J. Econ. Hist. 67, 128–159. https://doi.org/10.1017/S0022050707000058

Nordhaus, W.D., 1993. Rolling the “DICE”: an optimal transition path for controlling greenhouse gases. Resour. Energy Econ. 15, 27–50. https://doi.org/10.1016/0928-7655(93)90017-O

Nordhaus, W.D., 1973. The allocation of energy resources. Brookings Pap. Econ. Act. 1973, 529–576. https://doi.org/10.2307/2534202

Pfenninger, S., Hawkes, A., Keirstead, J., 2014. Energy systems modeling for twenty-first century energy challenges. Renew. Sustain. Energy Rev. 33, 74–86. https://doi.org/10.1016/j.rser.2014.02.003

Pfenninger, S., Hirth, L., Schlecht, I., Schmid, E., Wiese, F., Brown, T., Davis, C., Gidden, M., Heinrichs, H., Heuberger, C., Hilpert, S., Krien, U., Matke, C., Nebel, A., Morrison, R., Müller, B., Pleßmann, G., Reeg, M., Richstein, J.C., Shivakumar, A., Staffell, I., Tröndle, T., Wingenbach, C., 2018. Opening the black box of energy modelling: Strategies and lessons learned. Energy Strateg. Rev. 19, 73–81. https://doi.org/10.1016/j.esr.2017.12.002

Pindyck, R.S., 2013. Climate change policy: What do the models tell us? J. Econ. Lit. 51(3):860–72. https://doi.org/10.1257/jel.51.3.860

Pindyck, R.S., 2017. The use and misuse of models for climate policy. Rev. Environ. Econ. Policy 11(1):100–114. https://doi.org/10.1093/reep/rew012

Ringel, J.S., Eibner, C., Girosi, F., Cordova, A., McGlynn, E.A., 2010. Modeling health care policy alternatives. Health Serv. Res. 45, 1541–1558. https://doi.org/10.1111/j.1475-6773.2010.01146.x

Rivers, N., Jaccard, M., 2005. Combining top-down and bottom-up approaches to energy-economy modeling using discrete choice methods. Energy J. 26, 83–106. https://doi.org/10.5547/ISSN0195-6574-EJ-Vol26-No1-4

Rose, A.Z., Blomberg, S.B., 2010. Total economic consequences of terrorist attacks: Insights from 9/11. Peace Econ. Peace Sci. Public Policy 16, 1–14. https://doi.org/10.2202/1554-8597.1189

Sweeney, J.L., Weyant, J.P., 1979. The Energy Modeling Forum: Past, present and future. Energy Modeling Forum, Stanford University, EMF PP 6.1, Stanford, CA.

Tol, R.S.J., 2014. Ambiguity reduction by objective model selection, with an application to the costs of the EU 2030 climate targets. Energies 7, 6886–6896. https://doi.org/10.3390/en7116886

Trutnevyte, E., 2016. Does cost optimization approximate the real-world energy transition? Energy 106, 182–193. https://doi.org/10.1016/j.energy.2016.03.038

Weyant, J., 2017. Some contributions of integrated assessment models of global climate change. Rev. Environ. Econ. Policy 11(1):115–137. https://doi.org/10.1093/reep/rew018

Weyant, J., Kriegler, E., 2014. Preface and introduction to EMF 27. Clim. Change 123, 345–352. https://doi.org/10.1007/s10584-014-1102-7

Wing, I.S., 2009. Computable general equilibrium models for the analysis of energy and climate policies, in: Evans, J., Hunt, L. (Eds.), International Handbook on the Economics of Energy. Edward Elgar Publishing Ltd., pp. 332–366. https://doi.org/10.4337/9781849801997.00019