结直肠癌病理类型的多模态融合分类模型

Multimodal Fusion Classification Model for Colorectal Cancer Pathological Types

摘要: 结直肠癌发病率和死亡率持续上升，提高该疾病的诊断准确率成为重要议题。医学图像在诊断、治疗及健康管理等方面发挥着重要作用，深度学习技术在医学图像识别方面取得了显著进展，然而依靠单一的医学图像进行诊断则存在深层组织成像受限、人为判断差异导致等问题。深度学习结合多模态融合将成为未来的研究热点，而目前只有少数研究探索了将深度学习算法与组织病理学图像相结合的方向。本文构建了一种基于跨模态注意力机制的ResNet50双分支融合模型，将结直肠癌的组织病理切片图和MRI图进行多模态融合，以提升结直肠癌组织病理类型的识别准确率。该模型框架实现了多模态的中期和晚期融合，避免不同模态独有特征丢失，同时实现了不同模态间的高效信息交互。结果表明，进行多模态融合后，模型的整体分类性能、对关键类别的阳性样本识别能力、对9种不同类别的区分能力等都得到了提升(1.98%~14.70%)。t-SNE可视化也显示多模态融合模型在9种类别之间实现了更为有效地分离。该多模态融合模型有助于降低结直肠癌类别的误判和假阳性，有助于增强癌症区域的识别，模型及结果具有一定理论和应用价值。

Abstract: The incidence and mortality rates of colorectal cancer continue to rise, making improving the diagnostic accuracy of this disease a critical issue. Medical imaging plays a vital role in diagnosis, treatment, and health management, and deep learning has achieved significant progress in medical image recognition tasks. However, relying on a single type of medical image for diagnosis is limited by constraints such as restricted deep tissue imaging and inter-observer variability. Deep learning combined with multimodal fusion will emerge as a future research hotspot, while currently only a few studies have explored combining deep learning algorithms with histopathological images. This study constructs a ResNet50 dual-branch fusion model based on a cross-modal attention mechanism, integrating colorectal cancer histopathological section images and MRI features to enhance the accuracy of histopathological type recognition. The model framework implements multimodal mid-term and late fusion, avoiding the loss of modality-specific features while enabling efficient information interaction between different modalities. Results show that after multimodal fusion, the model’s overall classification performance, ability to identify positive samples of key categories, and capability to distinguish between 9 different categories were all improved (1.98%~14.70%). t-SNE visualization further demonstrated that the multimodal fusion model achieved more effective separation among the 9 categories. This multimodal fusion model helps reduce misdiagnosis and false positives of colorectal cancer pathological types, enhances the identification of cancerous regions, and holds theoretical and practical value.