基于跨模态注意力的皮肤病图像分割增强方法

Skin Lesion Image Segmentation Enhancement Method Based on Cross-Modal Attention

摘要: 精确的皮肤病灶分割在临床精准诊疗中至关重要。然而，现有方法主要依赖单模态影像数据，难以有效应对皮肤病灶形态的多样性和复杂病例中的语义模糊性问题。为此，本研究提出跨模态注意力引导的皮肤病灶分割网络(CMG-Net)，通过跨模态信息融合突破传统方法的性能瓶颈。该网络构建了跨模态数据协同机制，整合临床文本描述(包括病灶颜色、边界特征等语义信息)与视觉特征，实现跨模态信息的深度融合。并设计基于Transformer架构的跨模态特征融合模块(CMFM)，该模块通过双流交叉注意力机制实现视觉–语义特征的高效对齐与互补性交互。其中文本分支采用预训练语言模型提取深层语义表征，视觉分支通过动态参数共享策略实现模态特异性特征提取。在公开皮肤影像数据集ISIC2017上的实验结果表明，CMG-Net在复杂病例分割任务中显著优于现有单模态方法，尤其在形态相似病灶的鉴别任务中，IoU与Dice系数分别提升4.2%和4.3%。

Abstract: Accurate segmentation of skin lesions is crucial for clinical precision diagnosis and treatment. However, existing methods primarily rely on single-modal imaging data, which struggle to effectively address the diversity of skin lesion morphology and the semantic ambiguity in complex cases. To overcome these limitations, this study proposes a Cross-Modal Attention-Guided Skin Lesion Segmentation Network (CMG-Net), which breaks through the performance bottleneck of traditional methods by leveraging cross-modal information fusion. The network constructs a cross-modal data collaboration mechanism, integrating clinical textual descriptions (including semantic information such as lesion color and boundary features) with visual features to achieve deep fusion of cross-modal information. Additionally, a Transformer-based Cross-Modal Feature Fusion Module (CMFM) is designed, which utilizes a dual-stream cross-attention mechanism to enable efficient alignment and complementary interaction between visual and semantic features. In this module, the text branch employs a pre-trained language model to extract deep semantic representations, while the visual branch adopts a dynamic parameter-sharing strategy to achieve modality-specific feature extraction. Experimental results on the public skin imaging dataset ISIC2017 demonstrate that CMG-Net significantly outperforms existing single-modal methods in complex lesion segmentation tasks, particularly in distinguishing morphologically similar lesions, with improvements of 4.2% in IoU and 4.3% in Dice coefficient.