Persistently optimal policies in stochastic dynamic programming with generalized discounting

Jaśkiewicz, Anna and Matkowski, Janusz and Nowak, Andrzej S. (2011): Persistently optimal policies in stochastic dynamic programming with generalized discounting.

Abstract

In this paper we study a Markov decision process with a non-linear discount function. Our approach is in spirit of the von Neumann-Morgenstern concept and is based on the notion of expectation. First, we define a utility on the space of trajectories of the process in the finite and infinite time horizon and then take their expected values. It turns out that the associated optimization problem leads to a non-stationary dynamic programming and an infinite system of Bellman equations, which result in obtaining persistently optimal policies. Our theory is enriched by examples.

• Item Type: MPRA Paper
• Language: English
• Keywords: Stochastic dynamic programming, Persistently optimal policies, Variable discounting, Bellman equation, Resource extraction, Growth theory
• Subjects: D - Microeconomics > D9 - Intertemporal Choice > D90 - General; C - Mathematical and Quantitative Methods > C6 - Mathematical Methods ; Programming Models ; Mathematical and Simulation Modeling > C61 - Optimization Techniques ; Programming Models ; Dynamic Analysis

References

Becker, R.A., Boyd III, J.H., 1997. Capital Theory, Equilibrium Analysis and Recursive Utility. Blackwell Publishers, New York.

Berge, C., 1963. Topological Spaces. MacMillan, New York.

Bertsekas, D.P., 1977. Monotone mappings with application in dynamic programming, SIAM Journal on Control and Optimization 15, 438-464.

Bertsekas, D.P., Shreve, S.E., 1978. Stochastic Optimal Control: the Discrete Time Case. Academic Press, New York.

Blackwell, D., 1965. Discounted dynamic programming. Annals of Mathematical Statistics 36, 226-235.

Boyd III, J.H., 1990. Recursive utility and the Ramsey problem. Journal of Economic Theory 50, 326-345.

Brock, W.A., Mirman, L.J., 1972. Optimal economic growth and uncertainty: the discounted case. Journal of Economic Theory 4, 479-513.

Brown, L.D., Purves, R., 1973. Measurable selections of extrema. Annals of Statistics 1, 902-912.

Cho, Y., O'Regan, D., 2008. Fixed point theory for Volterra contractive operators of Matkowski type in Frechet spaces. Dynamics of Continuous, Discrete and Impulsive Systems, Series A: Mathematical Analysis 15, 871-884.

Denardo, E.V., 1967. Contraction mappings in the theory underlying dynamic programming. SIAM Review 9, 165-177.

Dubins, L.E., Savage, L.J., 1976. Inequalities for Stochastic Processes (How to Gamble if You Must). Dover. New York.

Feinberg, E.A., 2002. Total reward criteria. In: Feinberg, E.J., Shwartz, A. (Eds) Handbook of Markov decision processes: theory and methods, Kluwer Academic Publishers, Dordrecht, The Netherlands, 173-208.

Feinberg, E.A., Shwartz, A., (Eds) 2002. Handbook of Markov decision processes: theory and methods. Kluwer Academic Publishers, Dordrecht, The Netherlands.

Grandmont, J.M., 1977. Temporary general equilibrium theory. Econometrica 45, 535-572. Hicks, J.R., 1965. Capital and Growth. Oxford University Press, Oxford.

Hinderer, K., 1970. Foundations of Non-Stationary Dynamic Programming with Discrete Time Parameter. Lecture Notes in Operations Research 33, Springer-Verlag, NY.

Ja\'skiewicz, A., Matkowski, J., Nowak, A., 2011. On variable discounting in dynamic programming: applications to resource extraction and other economic models. Submitted.

Kertz, R.P., Nachman, D.C., 1979. Persistently optimal plans for nonstationary dynamic programming: the topology of weak convergence case. Annals of Probability 7, 811-826.

Koopmans, T.C., 1960. Stationary ordinal utility and impatience. Econometrica 28, 287-309.

Kreps, D.M., Porteus, E.L., 1978. Temporal resolution of uncertainty and dynamic choice theory. Econometrica 46, 185-200.

Lucas Jr., R.E., Stokey, N., 1984. Optimal growth with many consumers. Journal of Economic Theory 32, 139-171.

Matkowski, J., 1975. Integral solutions of functional equations. Dissertationes Mathematicae 127, 1-68.

Neveu, J., 1965. Mathematical Foundations of the Calculus of Probability. Holden-Day, San Francisco.

Nowak, A.S., 1986. Semicontinuous nonstationary stochastic games. Journal of Mathematical Analysis and Applications 117, 84-99.

Porteus, E., 1982. Conditions for characterizing the structure of optimal strategies in infinite-horizon dynamic programs. Journal of Optimization Theory and Applications 36, 419-431.

Samuelson, P., 1937. A note on measurement of utility. Review of Economic Studies 4, 155-161.

Schal M., 1975. Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal. Z Wahrsch verwandte Gebiete 32, 179-196.

Schal, M., 1975. On dynamic programming: compactness of the space of policies. Stochastic Processes and their Applications 3, 345-364.

Schal, M., 1981. Utility functions and optimal policies in sequential decision problems. In: Game Theory and Mathematical Economics (Moeschlin, O., Pallaschke, D., eds.), North-Holland, Amsterdam, 357-365.

Stokey, N.L., Lucas, Jr., R.E., Prescott, E., 1989. Recursive Methods in Economic Dynamics. Harvard University Press, Cambridge, MA.

Strauch, R., 1966. Negative dynamic programming. Annals of Mathematical Statistics 37, 871--890.