A natural language generation approach to support understanding and traceability of multi-dimensional preferential sensitivity analysis in multi-criteria decision making

Wulf, David and Bertsch, Valentin (2016): A natural language generation approach to support understanding and traceability of multi-dimensional preferential sensitivity analysis in multi-criteria decision making. Published in: Expert Systems with Applications , Vol. 83, (15 October 2017): pp. 131-144.

Abstract

Multi-Criteria Decision Analysis (MCDA) enables decision makers (DM) and decision analysts (DA) to analyse and understand decision situations in a structured and formalised way. With the increasing complexity of decision support systems (DSSs), it becomes challenging for both expert and novice users to understand and interpret the model results. Natural language generation (NLG) techniques are used in various DSSs to cope with this challenge as they reduce the cognitive effort to achieve understanding of decision situations. However, NLG techniques in MCDA have so far mainly been developed for deterministic decision situations or one-dimensional sensitivity analyses. In this paper, a concept for the generation of textual explanations for a multi-dimensional preferential sensitivity analysis in MCDA is developed. The key contribution is a NLG approach that provides detailed explanations of the implications of preferential uncertainties in Multi-Attribute Value Theory (MAVT). It generates a report that assesses the influences of simultaneous or separate variations of inter-criteria and intra-criteria preferential parameters determined within the decision analysis. We explore the added value of the natural language report in an online survey. Our results show that the NLG approach is particularly beneficial for difficult interpretational tasks.

• Item Type: MPRA Paper
• Language: English
• Keywords: Decision support systems; Multiple criteria analysis; Preferential uncertainty modelling; Natural language generation; Multi-dimensional preferential sensitivity analysis
• Subjects: C - Mathematical and Quantitative Methods > C6 - Mathematical Methods ; Programming Models ; Mathematical and Simulation Modeling; Q - Agricultural and Natural Resource Economics ; Environmental and Ecological Economics > Q4 - Energy > Q48 - Government Policy

References

Abu-Taha, R. (2011). Multi-criteria applications in renewable energy analysis: A literature review. paper presented at PICMET ‘11 Conference, July 31 - August 04, 2011, Portland. DOI: 10.1007/978-1-4471-5097-8_2.

Amgoud, L. and Prade, H. (2006). Explaining qualitative decision under uncertainty by argumentation. In Cohn, A. (Ed.), Proceeding AAAI 2006 proceedings of the 21st national conference on Artificial intelligence, Boston, MA, July 16-20, 2006, AAAI Press, Menlo Park, CA, pp. 219–224.

Bailey, D., Goonetilleke, A. and Deriche, M. (2011). A decision support system for site selection of large-scale infrastructure facilities using natural language. paper presented at PICMET ‘11 Conference, July 31 - August 04, 2011, Portland, available at: http://eprints.qut.edu.au/4223/ (accessed 12 January 2016).

Basson, L. and Petrie, J.G. (2007). An integrated approach for the consideration of uncertainty in decision making supported by Life Cycle Assessment. Environmental Modelling & Software, Vol. 22 No. 2, pp. 167–176. DOI: 10.1016/j.envsoft.2005.07.026.

Bélanger, M. and Martel, J.-M. (2005). An automated explanation approach for a decision support system based on MCDA. In Roth-Berghofer, T.R., Schulz, S. and Woody, A. (Eds.), Explanation-aware computing: Papers from the AAAI Fall Symposium, Arlington, VA, November 04-06, 2005, AAAI Press, Menlo Park, CA, pp. 21–34.

Bell, D.E. (1982). Regret in decision making under uncertainty. Operations research, Vol. 30 No. 5, pp. 961–981. DOI: 10.1287/opre.30.5.961.

Bell, M.L., Hobbs, B.F. and Ellis, H. (2003). The use of multi-criteria decision-making methods in the integrated assessment of climate change: Implications for IA practitioners. Socio-Economic Planning Sciences, Vol. 37 No. 4, pp. 289–316. DOI: 10.1016/S0038-0121(02)00047-2.

Belton, V. and Stewart, T.J. (2002), Multiple Criteria Decision Analysis - An Integrated Approach, Springer US, Boston.

Bertsch, V., Treitz, M., Geldermann, J. and Rentz, O. (2007). Sensitivity analyses in multi-attribute decision support for off-site nuclear emergency and recovery management. International Journal of Energy Sector Management, Vol. 1 No. 4, pp. 342-365.

Bertsch, V. (2008). Uncertainty Handling in Multi-Attribute Decision Support for Industrial Risk Management. University of Karlsruhe, Karlsruhe, 2008. DOI: 10.5445/KSP/1000007378.

Bertsch, V. and Fichtner, W. (2016). A participatory multi-criteria approach for power generation and transmission planning. Annals of Operations Research, Vol. 245 No. 1, pp. 177–207. DOI: 10.1007/s10479-015-1791-y.

Bishara, A.J. and Hittner, J.B. (2012). Testing the significance of a correlation with nonnormal data: Comparison of Pearson, Spearman, transformation, and resampling approaches. Psychological methods, Vol. 17 No. 3, pp. 399–417. DOI: 10.1037/a0028087.

Bloom, B.S., Krathwohl, D.R. and Masia, B.B. (1984), Taxonomy of educational objectives: The classification of educational goals Book 1: Cognitive Domain, Longman, New York.

Brans, J.-P. and Mareschal, B. (1994). The PROMCALC & GAIA decision support system for multicriteria decision aid. Decision Support Systems, Vol. 12 No. 4-5, pp. 297–310. DOI: 10.1016/0167-9236(94)90048-5.

Brehmer, B. (1980). In one word: Not from experience. Acta Psychologica, Vol. 45 No. 1-3, pp. 223–241. DOI: 10.1016/0001-6918(80)90034-7.

Broekhuizen, H., Groothuis-Oudshoorn, C.G.M., van Til, J.A., Hummel, J.M. and Jzerman, M.J. (2015). A review and classification of approaches for dealing with uncertainty in multi-criteria decision analysis for healthcare decisions. PharmacoEconomics, Vol. 33 No. 5, pp. 445–455. DOI: 10.1007/s40273-014-0251-x.

Browne, D., O’Regan, B. and Moles, R. (2010). Use of multi-criteria decision analysis to explore alternative domestic energy and electricity policy scenarios in an Irish city-region. Energy, Vol. 35 No. 2, pp. 518–528. DOI: 10.1016/j.energy.2009.10.020.

Buchanan, B.G. and Shortliffe, E.H. (1984), Rule-based expert systems: The MYCIN experiments of the Stanford Heuristic Programming Project, The Addison-Wesley series in artificial intelligence, Addison-Wesley, Reading, MA.

Butler, J., Jia, J. and Dyer, J. (1997). Simulation techniques for the sensitivity analysis of multi-criteria decision models. European Journal of Operational Research, Vol. 103 No. 3, pp. 531–546. DOI: 10.1016/S0377-2217(96)00307-4.

Carenini, G. and Moore, J.D. (2006). Generating and evaluating evaluative arguments. Artificial Intelligence, Vol. 170 No. 11, pp. 925–952. DOI: 10.1016/j.artint.2006.05.003.

Clark, A., Fox, C. and Lappin, S. (2010), The handbook of computational linguistics and natural language processing, Blackwell handbooks in linguistics, Wiley-Blackwell, Chichester, West Sussex, Malden, MA. DOI: 10.1002/9781444324044.

Cohen, J. (1988), Statistical power analysis for the behavioral sciences, 2nd ed., Lawrence Erlbaum Associates Inc., Hillsdale, N. J.

Corrente, S., French, S., Greco, S., Kadziński, M., Knowles, J.D., Mousseau, V., Siebert, J. and Słowiński, R. (2014). Drafting a Manifesto for DM-DSS Interaction (Working Group ‘DM Sense’). In Greco, S., Knowles, J.D., Miettinen, K. and Zitzler, E. (Eds.), Learning in Multiobjective Optimization: Report from Dagstuhl Seminar 2014, Dagstuhl Report, pp. 72–81.

Dhaliwal, J.S. and Benbasat, I. (1996). The Use and Effects of Knowledge-Based System Explanations: Theoretical Foundations and a Framework for Empirical Evaluation. Information Systems Research, Vol. 7 No. 3, pp. 342–362. DOI: 10.1287/isre.7.3.342.

Diakoulaki, D., Antunes, C.H. and Martins, A.G. (2005). MCDA and energy planning. In Figueira, J.R., Greco, S. and Ehrgott, M. (Eds.), Multiple criteria decision analysis: State of the art surveys, International Series in Operations Research & Management Science, Springer, New York, pp. 859–890. DOI: 10.1007/0-387-23081-5_21.

Durbach, I.N. and Stewart, T.J. (2009). Using expected values to simplify decision making under uncertainty. Omega, Vol. 37 No. 2, pp. 312–330. DOI: 10.1016/j.omega.2007.02.001.

Durbach, I.N. and Stewart, T.J. (2012). Modeling uncertainty in multi-criteria decision analysis. European Journal of Operational Research, Vol. 223 No. 1, pp. 1–14. DOI: 10.1016/j.ejor.2012.04.038.

Eisenführ, F., Weber, M. and Langer, T. (2010), Rational Decision Making, Springer, Berlin, London. DOI: 10.1007/978-3-642-02851-9.

Ellis, P.D. (2010). Effect sizes and the interpretation of research results in international business. Journal of International Business Studies, Vol. 41 No. 9, pp. 1581–1588. DOI: 10.1057/jibs.2010.39.

Ellis, P.D. (2013), The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results, 6th ed., Cambridge University Press, Cambridge. DOI: 10.1017/CBO9780511761676.

Fisher, R.A. (1922). On the Interpretation of X² from Contingency Tables, and the Calculation of P. Journal of the Royal Statistical Society, Vol. 85 No. 1, p. 87. DOI: 10.2307/2340521.

French, S. (1995). Uncertainty and Imprecision: Modelling and Analysis. The Journal of the Operational Research Society, Vol. 46 No. 1. DOI: 10.2307/2583837.

French, S., Maule, J. and Papamichail, K.N. (2009), Decision behaviour, analysis and support, Cambridge University Press, Cambridge, UK, New York.

Gardiner, P.C. and Edwards, W. (1975). Public values: Multiattribute-utility measurement for social decision making. In Kaplan, M.F. and Schwartz, S. (Eds.), Human Judgment and Decision Process, Academic Press Series in cognition and perception, Academic Press, New York, pp. 1–37. DOI: 10.1109/TSMC.1977.4309720.

Geldermann, J. (2010). Explanation Systems. In Ríos Insua, D. and French, S. (Eds.), E-Democracy: A group decision and negotiation practice, Advances in Group Decision and Negotiation, Vol. 5, Springer, Dordrecht, London, pp. 241–259.

Geldermann, J., Bertsch, V., Treitz, M., French, S., Papamichail, K.N. and Hämäläinen, R.P. (2009). Multi-criteria decision support and evaluation of strategies for nuclear remediation management. Omega, Vol. 37 No. 1, pp. 238–251. DOI: 10.1016/j.omega.2006.11.006.

Gilovich, T., Griffin, D.W. and Kahneman, D. (2002), Heuristics and biases: The psychology of intuitive judgement, Cambridge University Press, Cambridge, UK, New York.

Gosset, W.S. (1908). The probable error of a mean. Biometrika, Vol. 6 No. 1, pp. 1–25. DOI: 10.1093/biomet/6.1.1.

Graves, S.B. and Ringuest, J.L. (2009). Probabilistic dominance criteria for comparing uncertain alternatives: A tutorial. Omega, Vol. 37 No. 2, pp. 346–357. DOI: 10.1016/j.omega.2007.03.001.

Greco, S., Słowiński, R. and Zielniewicz, P. (2013). Putting Dominance-based Rough Set Approach and robust ordinal regression together. Decision Support Systems, Vol. 54 No. 2, pp. 891–903. DOI: 10.1016/j.dss.2012.09.013.

Greef, H.P.d. and Neerincx, M.A. (1995). Cognitive support: Designing aiding to supplement human knowledge. International Journal of Human-Computer Studies, Vol. 42 No. 5, pp. 531–571. DOI: 10.1006/ijhc.1995.1023.

Greening, L.A. and Bernow, S. (2004). Design of coordinated energy and environmental policies: Use of multi-criteria decision-making. Energy Policy, Vol. 32 No. 6, pp. 721–735. DOI: 10.1016/j.enpol.2003.08.017.

Greer, J.E., Falk, S., Greer, K.J. and Bentham, M.J. (1994). Explaining and justifying recommendations in an agriculture decision support system. Computers and Electronics in Agriculture, Vol. 11 No. 2-3, pp. 195–214. DOI: 10.1016/0168-1699(94)90008-6.

Gregor, S. and Benbasat, I. (1999). Explanations from Intelligent Systems: Theoretical Foundations and Implications for Practice. MIS Quarterly, Vol. 23 No. 4, pp. 497–530. DOI: 10.2307/249487.

Hämäläinen, R.P. and Alaja, S. (2008). The threat of weighting biases in environmental decision analysis. Ecological Economics, Vol. 68 No. 1-2, pp. 556–569. DOI: 10.1016/j.ecolecon.2008.05.025.

Hammond, K.R., Stewart, T.R., Brehmer, B. and Steinmann, D.O. (1975). Social Judgment Theory. In Kaplan, M.F. and Schwartz, S. (Eds.), Human Judgment and Decision Process, Academic Press Series in cognition and perception, Academic Press, New York.

Hauke, J. and Kossowski, T. (2011). Comparison of Values of Pearson’s and Spearman’s Correlation Coefficients on the Same Sets of Data. Quaestiones Geographicae, Vol. 30 No. 2, pp. 87–93. DOI: 10.2478/v10117-011-0021-1.

Henrion, M. and Druzdzel, M.J. (1991). Qualitative Propagation and Scenario-based Explanation of Probabilistic Reasoning. In Bonissone, P.P., Henrion, M., Kanal, L.N. and Lenner, J.F. (Eds.), Uncertainty and artificial intelligence, 6th ed., pp. 17–32.

Heo, E., Kim, J. and Boo, K.-J. (2010). Analysis of the assessment factors for renewable energy dissemination program evaluation using fuzzy AHP. Renewable and Sustainable Energy Reviews, Vol. 14 No. 8, pp. 2214–2220. DOI: 10.1016/j.rser.2010.01.020.

Hodgkin, J., Belton, V. and Koulouri, A. (2005). Supporting the intelligent MCDA user: A case study in multi-person multi-criteria decision support. European Journal of Operational Research, Vol. 160 No. 1, pp. 172–189. DOI: 10.1016/j.ejor.2004.03.007.

Hoffman, P.J., Earle, T.C. and Slovic, P. (1981). Multidimensional functional learning (MFL) and some new conceptions of feedback. Organizational Behavior and Human Performance, Vol. 27 No. 1, pp. 75–102. DOI: 10.1016/0030-5073(81)90040-4.

Hogarth, R.M. (1987), Judgement and choice: The psychology of decision, 2nd ed., Wiley, Chichester, New York.

Holtzman, S. (1988), Intelligent decision systems, Addison-Wesley Longman Publishing Co., Inc., Boston, MA.

Jessop, A. (2011). Using imprecise estimates for weights. Journal of the Operational Research Society, Vol. 62 No. 6, pp. 1048–1055.

Jessop, A. (2014). IMP: A decision aid for multiattribute evaluation using imprecise weight estimates. Omega, Vol. 49, pp. 18–29. DOI: 10.1016/j.omega.2014.05.001.

Jiménez, A., Mateos, A. and Ríos-Insua, S. (2005). Monte Carlo Simulation Techniques in a Decision Support System for Group Decision Making. Group Decision and Negotiation, Vol. 14 No. 2, pp. 109–130. DOI: 10.1007/s10726-005-2406-9.

Kadziński, M., Corrente, S., Greco, S. and Słowiński, R. (2014). Preferential reducts and constructs in robust multiple criteria ranking and sorting. OR Spectrum, Vol. 36 No. 4, pp. 1021–1053. DOI: 10.1007/s00291-014-0361-z.

Kahneman, D. and Knetsch, J.L. (1992). Valuing public goods: The purchase of moral satisfaction. Journal of Environmental Economics and Management, Vol. 22 No. 1, pp. 57–70. DOI: 10.1016/0095-0696(92)90019-S.

Kahneman, D., Slovic, P. and Tversky, A. (1982), Judgment under uncertainty: Heuristics and biases, Cambridge University Press, Cambridge, UK, New York.

Kass, R. and Finin, T. (1988). The Need for User Models in Generating Expert System Explanations. International Journal of Expert Systems - Special Issue: Natural Language and Expert Systems, Vol. 1 No. 4, pp. 345–375.

Kaya, T. and Kahraman, C. (2011). Multicriteria decision making in energy planning using a modified fuzzy TOPSIS methodology. Expert Systems with Applications, Vol. 38 No. 6, pp. 6577–6585. DOI: 10.1016/j.eswa.2010.11.081.

Keeney, R.L. and Raiffa, H. (1976), Decisions with multiple objectives: Preferences and value tradeoffs, Wiley, New York.

Kiker, G.A., Bridges, T.S., Varghese, A., Seager, T.P. and Linkov, I. (2005). Application of Multicriteria Decision Analysis in Environmental Decision Making. Integrated Environmental Assessment and Management, Vol. 1 No. 2, pp. 95–108. DOI: 10.1897/IEAM_2004a-015.1.

Kirkwood, C.W. (1997), Strategic decision making: Multiobjective decision analysis with spreadsheets, Duxbury Press, Belmont.

Kowalski, K., Stagl, S., Madlener, R. and Omann, I. (2009). Sustainable energy futures: Methodological challenges in combining scenarios and participatory multi-criteria analysis. European Journal of Operational Research, Vol. 197 No. 3, pp. 1063–1074. DOI: 10.1016/j.ejor.2007.12.049.

Labreuche, C., Maudet, N., Mousseau, V. and Ouerdane, W. (2012). Explanation of the robust additive preference model by even swap sequences. paper presented at 6th Multidisciplinary Workshop on Advances in Preference Handling, Montpellier.

Labreuche, C., Maudet, N. and Ouerdane, W. (2011). Minimal and Complete Explanations for Critical Multi-attribute Decisions. In Brafman, R.I., Roberts, F.S. and Tsoukiàs, A. (Eds.), Algorithmic decision theory, Piscataway, NJ, October 26-28, 2011, Springer, Berlin, Heidelberg, New York, pp. 121–134.

Lahdelma, R., Hokkanen, J. and Salminen, P. (1998). SMAA - Stochastic multiobjective acceptability analysis. European Journal of Operational Research, Vol. 106 No. 1, pp. 137–143. DOI: 10.1016/S0377-2217(97)00163-X.

Lahdelma, R. and Salminen, P. (2001). SMAA-2: Stochastic Multicriteria Acceptability Analysis for Group Decision Making. Operations Research, Vol. 49 No. 3, pp. 444–454.

Levy, H. (1992). Stochastic Dominance and Expected Utility: Survey and Analysis. Management Science, Vol. 38 No. 4, pp. 555–593.

Linkov, I., Varghese, A. and Jamil, S. (2004). Multi-criteria decision analysis: A framework for structuring remedial decisions at contaminated sites. In Linkov, I. and Ramadan, A.B. (Eds.), Comparative risk assessment and environmental decision making, Kluwer Academic Publishers, Boston, MA, pp. 15–54. DOI: 10.1007/1-4020-2243-3_2.

Loken, E. (2007). Use of multicriteria decision analysis methods for energy planning problems. Renewable and Sustainable Energy Reviews, Vol. 11 No. 7, pp. 1584–1595. DOI: 10.1016/j.rser.2005.11.005.

Loomes, G. and Sugden, R. (1982). Regret Theory: An Alternative Theory of Rational Choice under Uncertainty. Economic Journal, Vol. 92 No. 4, pp. 805–824. DOI: 10.2307/2232669.

Lühn, T., Schlömer, G., Schmidtmann, G., Lehde, B., Schmiesing, J., Hofmann, L. and Geldermann, J. (2014). Multi-Criteria Analysis of Grid Expansion Concepts on the Low Voltage Level. Zeitschrift für Energiewirtschaft, Vol. 38 No. 3, pp. 183–200. DOI: 10.1007/s12398-014-0134-z.

Mao, J.-Y. and Benbasat, I. (2000). The Use of Explanations in Knowledge-Based Systems: Cognitive Perspectives and a Process-Tracing Analysis. Journal of Management Information Systems, Vol. 17 No. 2, pp. 153–179. DOI: 10.1080/07421222.2000.11045646.

Mareschal, B. and Brans, J.-P. (1988). Geometrical representations for MCDA. European Journal of Operational Research, Vol. 34 No. 1, pp. 69–77. DOI: 10.1016/S0377-2217(00)00038-2.

Mateos, A., Jiménez, A. and Ríos-Insua, S. (2006). Monte Carlo simulation techniques for group decision making with incomplete information. European Journal of Operational Research, Vol. 3 No. 174, pp. 1842–1864. DOI: 10.1016/j.ejor.2005.02.057.

Matsatsinis, N.F. and Samaras, A.P. (2001). MCDA and preference disaggregation in group decision support systems. European Journal of Operational Research, Vol. 130 No. 2, pp. 414–429.

Mavrotas, G. and Trifillis, P. (2006). Multicriteria decision analysis with minimum information: Combining DEA with MAVT. Computers & Operations Research, Vol. 33 No. 8, pp. 2083–2098. DOI: 10.1016/j.cor.2004.11.023.

Morgan, M.G., Henrion, M. and Small, M. (1990), Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis, Cambridge University Press, Cambridge UK, New York.

Morton, A. and Fasolo, B. (2009). Behavioural decision theory for multi-criteria decision analysis: A guided tour. Journal of the Operational Research Society, Vol. 60 No. 2, pp. 268–275. DOI: 10.1057/palgrave.jors.2602550.

Mustajoki, J., Hämäläinen, R.P. and Salo, A. (2005). Decision Support by Interval SMART/SWING - Incorporating Imprecision in the SMART and SWING Methods. Decision Sciences, Vol. 36 No. 2, pp. 317–339. DOI: 10.1111/j.1540-5414.2005.00075.x.

Nunes, I., Miles, S., Luck, M. and Lucena, C. (2012). Investigating Explanations to Justify Choice. In Masthoff, J., Mobasher, B., Desmarais, M.C. and Nkambou, R. (Eds.), User modeling, adaptation, and personalization: 20th International Conference, UMAP 2012, Montreal, Canada, July 16-20, 2012. Proceedings, Information Systems and Applications, incl. Internet/Web, and HCI, Vol. 7379, Springer, Berlin, Heidelberg, pp. 212–224. DOI: 10.1007/978-3-642-31454-4_18.

Ouerdane, W., Maudet, N. and Tsoukiàs, A. (2010). Argumentation Theory and Decision Aiding. In Ehrgott, M., Figueira, J.R. and Greco, S. (Eds.), Trends in Multiple Criteria Decision Analysis, International Series in Operations Research & Management Science, Vol. 142, Springer US, pp. 177–208. DOI: 10.1007/978-1-4419-5904-1_7.

Pagano, R.R. (2013), Understanding statistics in the behavioral sciences, 10th ed., Wadsworth Publishing, Belmont, CA.

Papamichail, K.N. and French, S. (2000). Decision support in nuclear emergencies. Journal of Hazardous Materials, Vol. 71 No. 1-3, pp. 321–342. DOI: 10.1016/S0304-3894(99)00086-2.

Papamichail, K.N. and French, S. (2003). Explaining and justifying the advice of a decision support system - A natural language generation approach. Expert Systems with Applications, Vol. 24 No. 1, pp. 35–48. DOI: 10.1016/S0957-4174(02)00081-7.

Papamichail, K.N. and French, S. (2005). Design and evaluation of an intelligent decision support system for nuclear emergencies. Decision Support Systems, Vol. 41 No. 1, pp. 84–111. DOI: 10.1016/j.dss.2004.04.014.

Parikh, M., Fazlollahi, B. and Verma, S. (2001). The Effectiveness of Decisional Guidance: An Empirical Evaluation. Decision Sciences, Vol. 32 No. 2, pp. 303–332. DOI: 10.1111/j.1540-5915.2001.tb00962.x.

Pearson, K. (1900). On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philosophical Magazine Series 5, Vol. 50 No. 302, pp. 157–175. DOI: 10.1080/14786440009463897.

Pohekar, S.D. and Ramachandran, M. (2004). Application of multi-criteria decision making to sustainable energy planning - A review. Renewable and Sustainable Energy Reviews, Vol. 8 No. 4, pp. 365–381. DOI: 10.1016/j.rser.2003.12.007.

Reiter, E. and Dale, R. (1997). Building applied natural language generation systems. Natural Language Engineering, Vol. 3 No. 1, pp. 57–87. DOI: 10.1017/S1351324997001502.

Ren, J., Fedele, A., Mason, M., Manzardo, A. and Scipioni, A. (2013). Fuzzy Multi-actor Multi-criteria Decision Making for sustainability assessment of biomass-based technologies for hydrogen production. International Journal of Hydrogen Energy, Vol. 38 No. 22, pp. 9111–9120. DOI: 10.1016/j.ijhydene.2013.05.074.

Ribeiro, F., Ferreira, P. and Araújo, M. (2013). Evaluating future scenarios for the power generation sector using a Multi-Criteria Decision Analysis (MCDA) tool: The Portuguese case. Energy, Vol. 52, pp. 126–136. DOI: 10.1016/j.energy.2012.12.036.

Ríos Insua, D. and French, S. (1991). A framework for sensitivity analysis in discrete multi-objective decision-making. European Journal of Operational Research, Vol. 54 No. 2, pp. 176–190. DOI: 10.1016/0377-2217(91)90296-8.

Sánchez-Hernández, G. (2013). A contribution to the ranking and description of classifications. Dissertation, Instituto de Organización y Control de Sistemas Industriales, Universitat politécnica de Catalunya, Barcelona, 2013.

Scholten, L., Scheidegger, A., Reichert, P., Mauer, M. and Lienert, J. (2014). Strategic rehabilitation planning of piped water networks using multi-criteria decision analysis. Water research, Vol. 49, pp. 124–143. DOI: 10.1016/j.watres.2013.11.017.

Scholten, L., Schuwirth, N., Reichert, P. and Lienert, J. (2015). Tackling uncertainty in multi-criteria decision analysis – An application to water supply infrastructure planning. European Journal of Operational Research, Vol. 242 No. 1, pp. 243–260. DOI: 10.1016/j.ejor.2014.09.044.

Scott, J.A., Ho, W. and Dey, P.K. (2012). A review of multi-criteria decision-making methods for bioenergy systems. Energy, Vol. 42 No. 1, pp. 146–156. DOI: 10.1016/j.energy.2012.03.074.

Silver, M.S. (1991a). Decisional Guidance for Computer-Based Decision Support. MIS Quarterly, Vol. 15 No. 1, pp. 105–122. DOI: 10.2307/249441.

Silver, M.S. (1991b), Systems that support decision makers: Description and analysis, John Wiley information systems series, Wiley, Chichester, New York.

Spiegelhalter, D.J. and Knill-Jones, R.P. (1984). Statistical and Knowledge-Based Approaches to Clinical Decision-Support Systems, with an Application in Gastroenterology. Journal of the Royal Statistical Society. Series A (General), Vol. 147 No. 1, pp. 35–77. DOI: 10.2307/2981737.

Stewart, T.J. (1992). A critical survey on the status of multiple criteria decision making theory and practice. Omega, Vol. 20 No. 5-6, pp. 569–586. DOI: 10.1016/0305-0483(92)90003-P.

Stewart, T.J. (2005). Dealing with Uncertainties in MCDA. In Figueira, J.R., Greco, S. and Ehrgott, M. (Eds.), Multiple criteria decision analysis: State of the art surveys, International Series in Operations Research & Management Science, Vol. 78, Springer, New York, pp. 445–466. DOI: 10.1007/0-387-23081-5_11.

Streimikiene, D., Balezentis, T., Krisciukaitienė, I. and Balezentis, A. (2012). Prioritizing sustainable electricity production technologies: MCDM approach. Renewable and Sustainable Energy Reviews, Vol. 16 No. 5, pp. 3302–3311. DOI: 10.1016/j.rser.2012.02.067.

Swartout, W.R. and Moore, J.D. (1993). Explanation in Second Generation Expert Systems. In David, J.-M., Krivine, J.-P. and Simmons, R. (Eds.), Second generation expert systems, Springer Berlin Heidelberg, Berlin, pp. 543–585. DOI: 10.1007/978-3-642-77927-5_26.

Tervonen, T. (2014). JSMAA: Open source software for SMAA computations. International Journal of Systems Science, Vol. 45 No. 1, pp. 69–81. DOI: 10.1080/00207721.2012.659706.

Tervonen, T. and Figueira, J.R. (2008). A Survey on Stochastic Multicriteria Acceptability Analysis Methods. Journal of multi-criteria decision analysis, Vol. 15 No. 1-2, pp. 1–14. DOI: 10.1002/mcda.407.

Tintarev, N. and Masthoff, J. (2007). A Survey of Explanations in Recommender Systems. paper presented at IEEE 23rd International Conference on Data Engineering Workshop, April 15-20, 2007, Istanbul. DOI: 10.1109/ICDEW.2007.4401070.

Treitz, M. (2006), Production process design using multi-criteria analysis, Universitätsverlag Karlsruhe, Karlsruhe.

Tversky, A. (1972). Elimination by aspects: A theory of choice. Psychological Review, Vol. 79 No. 4, pp. 281–299. DOI: 10.1037/h0032955.

Walker, M.A., Whittaker, S.J., Stent, A., Maloor, P., Moore, J.D., Johnston, M. and Vasireddy, G. (2004). Generation and evaluation of user tailored responses in multimodal dialogue. Cognitive Science, Vol. 28 No. 5, pp. 811–840. DOI: 10.1207/s15516709cog2805_8.

Wang, J.-J., Jing, Y.-Y., Zhang, C.-F. and Zhao, J.-H. (2009). Review on multi-criteria decision analysis aid in sustainable energy decision-making. Renewable and Sustainable Energy Reviews, Vol. 13 No. 9, pp. 2263–2278. DOI: 10.1016/j.rser.2009.06.021.

Weiner, J.L. (1980). BLAH, a system which explains its reasoning. Artificial Intelligence, Vol. 15 No. 1-2, pp. 19–48. DOI: 10.1016/0004-3702(80)90021-1.

Wybo, J.-L. (2006). Editorial. International Journal of Emergency Management, Vol. 3 No. 2/3, pp. 99-100.

Ye, R.L. (1995). The value of explanation in expert systems for auditing: An experimental investigation. Expert Systems with Applications, Vol. 9 No. 4, pp. 543–556. DOI: 10.1016/0957-4174(95)00023-2.

Zadeh, L.A. (1965). Fuzzy sets. Information and Control, Vol. 8 No. 3, pp. 338–353. DOI: 10.1016/S0019-9958(65)90241-X.

Zhou, P., Ang, B.W. and Poh, K.L. (2006). Decision analysis in energy and environmental modeling: An update. Energy, Vol. 31 No. 14, pp. 2604–2622. DOI: 10.1016/j.energy.2005.10.023.

Zimmermann, H.-J. (2000). An application-oriented view of modeling uncertainty. European Journal of Operational Research, Vol. 122 No. 2, pp. 190–198. DOI: 10.1016/S0377-2217(99)00228-3.