RTRFNet：面向RGB-T分割的鲁棒融合网络——应对传感器模态缺失的鲁棒性研究

RTRFNet: A Robust Fusion Network for RGB-T Segmentation—Robustness Research against Missing Sensor Modalities

摘要: RGB-热红外(RGB-T)语义分割对于在弱光或黑暗等复杂环境中运行的机器人系统至关重要。然而，传统的多模态融合方法往往导致不同模态特征的高度耦合，使得模型在真实场景下遭遇传感器信号缺失时极度脆弱，性能发生严重退化。为此，本文提出了RTRFNet，一个基于教师–学生学习机制的多模态鲁棒网络，从根本上打破了推理阶段对双源输入的强依赖。在训练阶段，网络通过一个精简的通道注意力特征融合模块(CA-FFM)高效聚合跨模态互补线索，为中枢的轻量化感知头(教师分支)构建完备的联合语义表示；随后，引入多模态知识蒸馏(MKD)策略，利用教师分支输出的高质量软分布，隐式地监督并引导完全独立的RGB与热红外双流网络(学生分支)，促使其充分获得并内化跨模态的丰富上下文知识。这种联合训练机制赋予了系统在推理阶段极高的灵活性：系统既能在全模态下移除教师网络并执行极具参数效率的决策层均值融合，也能在单传感器失效时仅激活存活链路进行高精度的独立推理。在主流基准数据集上的大量实验证明，RTRFNet不仅维持了全模态下的前沿精度，更在单模态缺失的极端条件下展现出了卓越的鲁棒性与轻量化部署优势。

Abstract: RGB-Thermal (RGB-T) semantic segmentation is crucial for robotic systems operating in complex environments such as low-light or dark conditions. However, traditional multimodal fusion methods often lead to highly coupled modal features, making models extremely vulnerable to severe performance degradation when encountering missing sensor signals in real-world scenarios. To address this, this paper proposes RTRFNet, a robust multimodal network based on a teacher-student learning mechanism, which fundamentally breaks the strong reliance on dual-source inputs during the inference phase. During training, the network efficiently aggregates cross-modal complementary cues through a lightweight Channel Attention Feature Fusion Module (CA-FFM) to build a comprehensive joint semantic representation for a central lightweight perception head (teacher branch). Subsequently, a Multimodal Knowledge Distillation (MKD) strategy is introduced. It utilizes the high-quality soft distributions output by the teacher branch to implicitly supervise and guide the completely independent RGB and thermal dual-stream networks (student branches), prompting them to acquire and internalize rich cross-modal contextual knowledge. This joint training mechanism endows the system with extremely high flexibility during inference: by removing the teacher network, the system can perform highly parameter-efficient decision-level mean fusion under full modalities, or solely activate the surviving link for high-precision independent inference when a single sensor fails. Extensive experiments on mainstream benchmark datasets demonstrate that RTRFNet not only maintains state-of-the-art accuracy under full modalities but also exhibits exceptional robustness and lightweight deployment advantages under extreme missing modality conditions.