基于生存函数的连续时间强化学习问题

Continuous-Time Reinforcement Learning Problem Based on Survival Functions

摘要: 在智能体具有时间不一致性偏好的条件下，用生存分析中的生存函数和风险率刻画折现函数，使用跳跃–扩散过程对强化学习中的环境状态建模，探讨跳跃过程对值函数和偏好逆转时间的影响。研究表明：1) 跳跃–扩散过程下值函数与时间高度相关，且发生偏好逆转的时间会提前。2) 跳跃幅度服从同一分布时，随着泊松过程的强度增大，值函数的值也会越大，发生偏好逆转的时间会越早。3) 在同一泊松过程强度下，当跳跃幅度服从正态分布时，值函数的值最大，发生偏好逆转的时间最早。这些结论为决策者预测识别潜在的突发事件，从而采取相应的预防措施提供重要参考。

Abstract: Under the condition of agents having time-inconsistent preferences, this study characterizes the discount function using survival functions and hazard rates from survival analysis. It models the environmental state in reinforcement learning using jump-diffusion processes and explores the impact of jump processes on value functions and the time of preference reversal. The study in-dicates that: 1) Under the jump-diffusion process, value functions are highly correlated with time, and the time of preference reversal is advanced. 2) When jump amplitudes follow the same distribution, with an increase in the intensity of the Poisson process, the value of the value func-tion also increases, and the time of preference reversal occurs earlier. 3) Under the same inten-sity of the Poisson process, when jump amplitudes follow a normal distribution, the value of the value function is maximized, and the time of preference reversal is earliest. These findings pro-vide important references for decision-makers to predict and identify potential sudden events and take corresponding preventive measures.