Jaśkiewicz, Anna and Matkowski, Janusz and Nowak, Andrzej S. (2011): Persistently optimal policies in stochastic dynamic programming with generalized discounting.

PDF
MPRA_paper_31755.pdf Download (164kB)  Preview 
Abstract
In this paper we study a Markov decision process with a nonlinear discount function. Our approach is in spirit of the von NeumannMorgenstern concept and is based on the notion of expectation. First, we define a utility on the space of trajectories of the process in the finite and infinite time horizon and then take their expected values. It turns out that the associated optimization problem leads to a nonstationary dynamic programming and an infinite system of Bellman equations, which result in obtaining persistently optimal policies. Our theory is enriched by examples.
Item Type:  MPRA Paper 

Original Title:  Persistently optimal policies in stochastic dynamic programming with generalized discounting 
English Title:  Persistently Optimal Policies in Stochastic Dynamic Programming with Generalized Discounting 
Language:  English 
Keywords:  Stochastic dynamic programming, Persistently optimal policies, Variable discounting, Bellman equation, Resource extraction, Growth theory 
Subjects:  D  Microeconomics > D9  Intertemporal Choice > D90  General C  Mathematical and Quantitative Methods > C6  Mathematical Methods ; Programming Models ; Mathematical and Simulation Modeling > C61  Optimization Techniques ; Programming Models ; Dynamic Analysis 
Item ID:  31755 
Depositing User:  Andrzej Nowak 
Date Deposited:  21 Jun 2011 20:29 
Last Modified:  26 Aug 2016 12:00 
References:  Becker, R.A., Boyd III, J.H., 1997. Capital Theory, Equilibrium Analysis and Recursive Utility. Blackwell Publishers, New York. Berge, C., 1963. Topological Spaces. MacMillan, New York. Bertsekas, D.P., 1977. Monotone mappings with application in dynamic programming, SIAM Journal on Control and Optimization 15, 438464. Bertsekas, D.P., Shreve, S.E., 1978. Stochastic Optimal Control: the Discrete Time Case. Academic Press, New York. Blackwell, D., 1965. Discounted dynamic programming. Annals of Mathematical Statistics 36, 226235. Boyd III, J.H., 1990. Recursive utility and the Ramsey problem. Journal of Economic Theory 50, 326345. Brock, W.A., Mirman, L.J., 1972. Optimal economic growth and uncertainty: the discounted case. Journal of Economic Theory 4, 479513. Brown, L.D., Purves, R., 1973. Measurable selections of extrema. Annals of Statistics 1, 902912. Cho, Y., O'Regan, D., 2008. Fixed point theory for Volterra contractive operators of Matkowski type in Frechet spaces. Dynamics of Continuous, Discrete and Impulsive Systems, Series A: Mathematical Analysis 15, 871884. Denardo, E.V., 1967. Contraction mappings in the theory underlying dynamic programming. SIAM Review 9, 165177. Dubins, L.E., Savage, L.J., 1976. Inequalities for Stochastic Processes (How to Gamble if You Must). Dover. New York. Feinberg, E.A., 2002. Total reward criteria. In: Feinberg, E.J., Shwartz, A. (Eds) Handbook of Markov decision processes: theory and methods, Kluwer Academic Publishers, Dordrecht, The Netherlands, 173208. Feinberg, E.A., Shwartz, A., (Eds) 2002. Handbook of Markov decision processes: theory and methods. Kluwer Academic Publishers, Dordrecht, The Netherlands. Grandmont, J.M., 1977. Temporary general equilibrium theory. Econometrica 45, 535572. Hicks, J.R., 1965. Capital and Growth. Oxford University Press, Oxford. Hinderer, K., 1970. Foundations of NonStationary Dynamic Programming with Discrete Time Parameter. Lecture Notes in Operations Research 33, SpringerVerlag, NY. Ja\'skiewicz, A., Matkowski, J., Nowak, A., 2011. On variable discounting in dynamic programming: applications to resource extraction and other economic models. Submitted. Kertz, R.P., Nachman, D.C., 1979. Persistently optimal plans for nonstationary dynamic programming: the topology of weak convergence case. Annals of Probability 7, 811826. Koopmans, T.C., 1960. Stationary ordinal utility and impatience. Econometrica 28, 287309. Kreps, D.M., Porteus, E.L., 1978. Temporal resolution of uncertainty and dynamic choice theory. Econometrica 46, 185200. Lucas Jr., R.E., Stokey, N., 1984. Optimal growth with many consumers. Journal of Economic Theory 32, 139171. Matkowski, J., 1975. Integral solutions of functional equations. Dissertationes Mathematicae 127, 168. Neveu, J., 1965. Mathematical Foundations of the Calculus of Probability. HoldenDay, San Francisco. Nowak, A.S., 1986. Semicontinuous nonstationary stochastic games. Journal of Mathematical Analysis and Applications 117, 8499. Porteus, E., 1982. Conditions for characterizing the structure of optimal strategies in infinitehorizon dynamic programs. Journal of Optimization Theory and Applications 36, 419431. Samuelson, P., 1937. A note on measurement of utility. Review of Economic Studies 4, 155161. Schal M., 1975. Conditions for optimality in dynamic programming and for the limit of nstage optimal policies to be optimal. Z Wahrsch verwandte Gebiete 32, 179196. Schal, M., 1975. On dynamic programming: compactness of the space of policies. Stochastic Processes and their Applications 3, 345364. Schal, M., 1981. Utility functions and optimal policies in sequential decision problems. In: Game Theory and Mathematical Economics (Moeschlin, O., Pallaschke, D., eds.), NorthHolland, Amsterdam, 357365. Stokey, N.L., Lucas, Jr., R.E., Prescott, E., 1989. Recursive Methods in Economic Dynamics. Harvard University Press, Cambridge, MA. Strauch, R., 1966. Negative dynamic programming. Annals of Mathematical Statistics 37, 871890. 
URI:  https://mpra.ub.unimuenchen.de/id/eprint/31755 