基于RGB-D联合语义分割和边界检测的跨模态融合网络

Cross-Modal Fusion Network for RGB-D Joint Semantic Segmentation and Boundary Detection

摘要: 语义分割和边界检测是自动驾驶汽车实现准确环境感知的两大关键任务，然而现有研究多将两者视为独立的任务，或者将语义和边界特征进行简单堆叠，忽略了两者之间的内在联系，缺乏对物体与边界间依赖关系的显式建模，易导致在颜色相近区域出现边界模糊、类别混淆。为此，本文提出了一个跨模态联合感知网络，通过引入深度信息Depth为RGB图像提供几何先验，并建立了一种动态边界引导机制，利用边界信息与几何结构共同指导语义分割过程。具体来说，网络采用双分支结构分别捕获RGB信息、Depth信息并提出了一个边界引导的跨模态融合模块BGCF (Boundary-Guided Cross-modality Fusion Module)，通过动态融合不同层级的RGB特征与深度特征，建立二者之间的全局依赖关系，从而获取更准确的多级融合特征。为进一步捕获多尺度全局信息，本文引用了自适应金字塔上下文模块APC (Adaptive Pyramid Context Module)。在解码阶段，采用两个独立的解码器，语义解码器通过BGCF模块输出精确的分割结果，边界解码器则采用残差结构有效融合局部细节与全局信息，提升边界检测准确性。实验结果表明，该方法在Cityscapes数据集上取得了优越的分割精度与边界检测精度。

Abstract: Semantic segmentation and boundary detection are two critical tasks for autonomous vehicles to achieve precise environmental awareness. However, most existing methods treat these tasks as independent or merely stack semantic and boundary features, neglecting the intrinsic relationship between them. This oversight results in a lack of explicit modeling of the interdependence between objects and boundaries, often leading to blurry boundaries and category confusion in regions with similar colors. To address this issue, we propose a cross-modal joint perception network, which enhances RGB images by incorporating depth information as geometric priors. Additionally, the method establishes a dynamic boundary guidance mechanism that utilizes both boundary information and geometric structure to jointly steer the semantic segmentation process. Specifically, the method employs a dual-branch architecture to separately capture RGB information and Depth information while introducing a Boundary Guided Cross Modality Fusion Module (BGCF). By dynamically fusing RGB features and depth features at various levels, we establish a global dependency relationship between the two modalities to obtain more accurate multi-level fusion features. To further enhance the capture of multi-scale global information, this paper references the Adaptive Pyramid Context Module (APC). In the decoding stage, two independently designed decoders are used. One for semantic output that generates precise segmentation results through the BGCF, another for boundary detection that employs lightweight residual units to effectively integrate local details with global context, improving boundary detection accuracy. Experimental results demonstrate that the method achieves superior segmentation and boundary detection accuracy on the Cityscapes dataset.