基于分层策略与世界模型的多智能体深度确定性策略梯度算法

Multi-Agent Deep Deterministic Policy Gradient Algorithm Based on Hierarchical Strategies and World Models

摘要: 针对三维环境中多无人机路径规划面临着样本效率低、长时程决策困难和鲁棒性不足等挑战，本文提出一种基于分层策略与世界模型增强的多智能体深度确定性策略梯度算法框架(HWC-MADDPG)。首先，引入对比学习机制，从高维观测中提取时序一致性的鲁棒状态表征，增强了状态表征的区分度；其次，设计多智能体层次化策略网络架构，通过高层策略网络规划宏观意图，低层策略网络执行具体动作的方式，将路径规划任务分解，提升决策能力；最后，集成共享的世界模型，通过其内在的前瞻性推演生成想象奖励，优化Critic网络的价值评估，提升了决策前瞻性和收敛速度。实验结果表明，本文提出的算法在学习速度、策略稳定性和飞行安全性上均优于传统的多智能体深度确定性策略梯度算法(MADDPG)。该研究为解决三维环境下的多智能体路径规划问题提供了一种更高效的解决方案，具有一定的理论价值与应用前景。

Abstract: Addressing challenges in multi-UAV path planning within 3D environments—such as low sample efficiency, difficulties in long-term decision-making, and insufficient robustness—this paper proposes a hierarchical strategy and world model-enhanced multi-agent deep deterministic policy gradient algorithm framework (HWC-MADDPG). First, a contrastive learning mechanism is introduced to extract temporally consistent robust state representations from high-dimensional observations, enhancing the discriminative power of state representations. Second, a hierarchical multi-agent policy network architecture is designed. By decomposing the path planning task—where the high-level policy network formulates macro-intentions and the low-level policy network executes specific actions—decision-making capabilities are enhanced. Finally, an integrated shared world model generates imagined rewards through its inherent forward-looking inference, optimizing the value assessment of the Critic network and improving decision foresight and convergence speed. Experimental results demonstrate that the proposed algorithm outperforms the traditional Multi-Agent Deep Deterministic Policy Gradient (MADDPG) in learning speed, policy stability, and flight safety. This research offers a more efficient solution for multi-agent path planning in 3D environments, holding significant theoretical value and practical application potential.