融合CT影像与文本信息的胰腺疾病深度学习诊断方法研究

Deep Learning-Based Diagnosis of Pancreatic Diseases via Fusion of CT Images and Textual Information

摘要: 高分辨增强CT是胰腺疾病评估的首选影像手段，但胰腺病异质性高、阅片负荷大，易致漏误诊。本研究提出轻量级端到端多模态网络eeMulNet，实现胰腺病变快速诊断。首先，以nnU-Net为骨干，在自适应3D编码–解码框架下完成胰腺分割；继而，用轻量3D-MobileNetV2提取胰腺影像特征；同时，以BERT嵌入患者临床文本，生成语义向量，影像与文本特征融合后再次输入MobileNetV2，建立eeMulNet完成病变分类。单中心回顾性实验中模型准确率达100%，单例推理时间 < 0.05 s。本研究提出的eeMulNet可在保持低计算量的情况下实现胰腺CT病变快速、精准检测。

Abstract: High-resolution contrast-enhanced CT is the first-line imaging modality for pancreatic diseases; however, lesion heterogeneity and heavy reading workload frequently lead to missed or misdiagnoses of pancreatic diseases. We propose a lightweight end-to-end multimodal network, termed eeMulNet, that achieves the diagnosis of pancreatic diseases. First, a self-configuring 3D encoder-decoder built on nnU-Net performs automatic pancreas segmentation. Next, a lightweight 3D-MobileNetV2 extracts imaging features from the segmented volume, while demographic and clinical free-text are simultaneously embedded by BERT to generate semantic vectors. Fused image-text features are then forwarded through a second MobileNetV2, yielding the final lesion classification. In a single-center retrospective study, eeMulNet achieved 100% accuracy, and an average inference time of <0.05 s per case. The current study demonstrates that eeMulNet enables rapid and accurate detection of pancreatic CT lesions with minimal computational cost on CT.