基于Transformer-CNN的红外与可见光融合网络

Infrared and Visible Image Fusion Network Based on Transformer-CNN

摘要: 红外与可见光图像融合是多模态融合领域的重要研究方向之一，广泛应用于医药、工业、自动驾驶等领域。本文提出了一种基于Transformer-CNN网络的融合算法。首先，构建双分支特征提取网络提取不同模态图像的特征信息；然后，在融合阶段通过模态间特征差分计算提取互补特征，采用参数自适应的Swish激活函数动态生成通道权重，结合全局平均池化压缩与特征级联策略，实现红外与可见光模态的跨尺度特征融合。最后，提出一种掩码损失以提升融合质量。在公开数据集上的实验结果表明，该方法在进行红外和可见光图像融合后图像具有清晰的背景纹理细节。在主观和客观评价上均能取得较好或相近的结果。

Abstract: Infrared and visible image fusion is a significant research direction in multimodal fusion, with extensive applications in medicine, industry, autonomous driving, and other fields. This paper proposes a fusion algorithm based on a Transformer-CNN hybrid network. First, a dual-branch feature extraction network is constructed to extract features from different modal images. Subsequently, during the fusion stage, complementary features are extracted through cross-modal feature difference computation. A parameter-adaptive Swish activation function is employed to dynamically generate channel weights, which are combined with global average pooling for feature compression and cascading strategies, enabling cross-scale feature fusion of infrared and visible modalities. Finally, a mask loss is introduced to enhance fusion quality. Experimental results on public datasets demonstrate that the fused images produced by this method exhibit clear background texture details. Both subjective evaluation and objective metrics indicate that our approach achieves superior or comparable results.