基于Actor-Critic强化学习的投资与消费问题

Investment and Consumption Problems Based on Actor-Critic Reinforcement Learning

摘要: 本文研究了基于Actor-Critic强化学习的最优资产与消费问题。为了描述个体对退休后实际消费水平较低的现象，我们假设个体在退休后的最低消费水平和养老金水平较低，金融市场的资产价格由马尔可夫链调节，考虑通胀因素、习惯消费水平，建立状态转换的财富模型。利用动态规划原理得到了Hamilton-Jacobi-Bellman(HJB)方程。由于扩散过程和状态切换，几乎不可能得到一个封闭形式的解。我们设计出一种基于Actor-Critic强化学习框架下的数值算法来解决最优控制问题，通过对财富过程、优化函数的离散化和对值函数、控制函数的神经网络参数化，采用策略梯度下降算法来改进控制函数，而对于值函数，采用一种TD误差方法来更新。最后是对该优化问题的数值结果展示。

Abstract: This paper investigates the optimal asset and consumption problem based on Actor-Critic reinforcement learning. To describe the phenomenon of relatively low actual consumption levels after retirement, we assume that individuals have lower minimum consumption levels and pension levels after retirement. The asset prices in the financial market are regulated by a Markov chain, and we consider inflation factors and habitual consumption levels to establish a wealth model with state transitions. By applying the principle of dynamic programming, we derive the Hamilton-Jacobi-Bellman (HJB) equation. Due to the diffusion process and state switching, it is nearly impossible to obtain a closed-form solution. We design a numerical algorithm based on the Actor-Critic reinforcement learning framework to solve the optimal control problem. By discretizing the wealth process and the optimization function, and parameterizing the value function and control function using neural networks, we use a gradient descent algorithm to improve the control function. For the value function, we use a TD error method to update it. Finally, we present the numerical results of the optimization problem.