面向医疗视觉问答的动态问题编码与知识对比推理模型

Dynamic Question Encoding and Knowledge Contrastive Reasoning Model for Medical Visual Question Answering

摘要: 医疗视觉问答(Med-VQA)旨在基于医学图像与自然语言问题预测可信且准确答案。现有方法主要依赖医学图像特征解析，缺乏对问题语义的深入建模，未能充分考虑开放式与封闭式问题在语义理解上的差异需求。此外，面对多义性强、语境依赖性高的医疗问题，文本查询往往缺乏足够的描述性内容，仅依赖图像和文本特征的融合会导致跨模态对齐不足。针对上述问题，本文提出动态问题编码与知识对比推理(DKCR)模型。该模型利用动态问题编码模块根据问题类型动态建模，既增强开放式问题的语义表征，又避免对封闭式问题引入冗余特征。为减轻跨模态语义偏差，DKCR的知识对比学习通过知识–图像与知识–问题两条路径联合约束潜在表征空间，促进跨模态特征的一致性与互补性。同时，引入外部知识以弥补查询语义不足，丰富问题表征，并通过知识驱动的引导注意力机制，实现视觉、文本与知识模态的深度交互与细粒度对齐。在VQA-RAD与SLAKE数据集上的实验结果表明，DKCR在整体性能上优于现有方法，消融研究进一步验证了各模块的独立价值与协同增益。

Abstract: Medical Visual Question Answering (Med-VQA) is a task that aims to predict reliable and accurate answers based on medical images and natural language questions. Existing methods primarily rely on the analysis of medical image features, lacking in-depth modeling of question semantics and failing to fully consider the distinct semantic understanding requirements of open-ended and closed-ended questions. Furthermore, medical questions often exhibit strong ambiguity and high context dependence, where textual queries frequently lack sufficient descriptive content. Solely relying on the fusion of image and text features leads to insufficient cross-modal alignment. To address these issues, this paper proposes a Dynamic Question Encoding and Knowledge Contrastive Reasoning Model (DKCR) model. This model employs a dynamic question encoding module to adaptively model questions based on their types, enhancing semantic representation for open-ended questions while avoiding the introduction of redundant features for closed-ended ones. To mitigate cross-modal semantic bias, DKCR’s knowledge contrastive learning constrains the latent representation space through two pathways—knowledge-image and knowledge-question—promoting consistency and complementarity of cross-modal features. Concurrently, external knowledge is incorporated to compensate for insufficient query semantics and enrich question representation. A knowledge-driven co-attention mechanism facilitates deep interaction and fine-grained alignment among visual, textual, and knowledge modalities. Experimental results on the VQA-RAD and SLAKE datasets demonstrate that DKCR outperforms existing methods in overall performance, and ablation studies further validate the individual value and synergistic benefits of each module.