基于交叉记忆注意力的视觉问答模型
Visual Question Answering Model Based on Cross Memory Attention
DOI: 10.12677/CSA.2023.136116, PDF,    国家科技经费支持
作者: 文必成, 常青玲, 刘兴林, 崔 岩*：五邑大学智能制造学部，广东 江门
关键词: 视觉问答；注意力机制；特征融合；Visual Question Answering； Attention Mechanism； Feature Fusion
摘要: 视觉问答是一项涉及图像和文本的多模态任务，给定一个图像和一个用自然语言表达的问题，视觉问答系统需要对视觉和文本信息同时进行复杂的理解，提供关于图像的这个问题的准确答案。现有的视觉问答模型在获取与问题相关的图像区域时，不能有效利用文本与图像信息的多层次特征信息，因此，我们使用自注意记忆层，使得所得特征的每一层包含之前的先验知识。同时利用交叉记忆模块，在解码端的所有引导注意力层中输入编码端的各级加权特征，通过引导注意力，融合低层次与高层次信息，使用多层次信息更好地关注图像特征中的关键区域。本文在VQA v2.0数据集上进行了对比实验，表明该模型能充分利用图像和文本的多层次特征信息，与当前主流模型相比更具优越性。
Abstract: Visual question answering (VQA) is a multimodal task involving images and text. Given an image and a question expressed in natural language, a visual question answering system needs to under-stand both visual and textual information in a complex way to provide an accurate answer to this question about the image. The existing visual question answering model cannot effectively utilize the multi-level feature information of text and image information when acquiring the image area related to the question. Therefore, we use the self-attention memory layer so that each layer of the obtained feature contains the previous prior knowledge. At the same time, the cross-memory module is used to input the weighted features of the encoder at all levels in all the guided attention layers of the decoder. By guided attention, the low-level and high-level information are fused, and the multi-level information is used to better focus on key areas in the image features. In this paper, we conduct comparative experiments on the VQA v2.0 dataset and show that the model can make full use of the multi-level feature information of images and text, and is superior to the current main-stream models.
文章引用：文必成, 常青玲, 刘兴林, 崔岩. 基于交叉记忆注意力的视觉问答模型[J]. 计算机科学与应用, 2023, 13(6): 1188-1198. https://doi.org/10.12677/CSA.2023.136116

参考文献

[1] Stahlschmidt, S.R., Ulfenborg, B. and Synnergren, J. (2022) Multimodal Deep Learning for Biomedical Data fusion: A Review. Briefings in Bioinformatics, 23, bbab569. [Google Scholar] [CrossRef] [PubMed]
[2] Antol, S., Agrawal, A., Lu, J., et al. (2015) VQA: Visual Question Answering. Proceedings of the 2015 IEEE International Conference on Computer Vision, Santiago, 7-13 December 2015, 2425-2433. [Google Scholar] [CrossRef] 
[3] Sharma, H. and Jalal, A.S. (2021) A Survey of Methods, Datasets and Evaluation Metrics for Visual Question Answering. Image and Vision Computing, 116, Article ID: 104327. [Google Scholar] [CrossRef] 
[4] Vu, M.H., Löfstedt, T., Nyholm, T., et al. (2020) A Ques-tion-Centric Model for Visual Question Answering in Medical Imaging. IEEE Transactions on Medical Imaging, 39, 2856-2868. [Google Scholar] [CrossRef] 
[5] Zhang, D., Cao, R. and Wu, S. (2019) Information Fusion in Visual Question Answering: A Survey. Information Fusion, 52, 268-280. [Google Scholar] [CrossRef] 
[6] Albawi, S., Mohammed, T.A. and Al-Zawi, S. (2017) Under-standing of a Convolutional Neural Network. Proceedings of 2017 International Conference on Engineering and Tech-nology (ICET), Antalya, 21-23 August 2017, 1-6. [Google Scholar] [CrossRef] 
[7] Hochreiter, S. and Schmidhuber, J. (1997) Long Short-Term Memory. Neural Computation, 9, 1735-1780. [Google Scholar] [CrossRef] [PubMed]
[8] Anderson, P., He, X., Buehler, C., et al. (2018) Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering. Proceedings of the 2018 IEEE/CVF Con-ference on Computer Vision and Pattern Recognition, Lake City, 18-23 June 2018, 6077-6086. [Google Scholar] [CrossRef] 
[9] Ren, S., He, K., Girshick, R. and Sun, J. (2017) Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Ma-chine Intelligence, 39, 1137-1149. [Google Scholar] [CrossRef] 
[10] Nam, H., Ha, J.W. and Kim, J. (2017) Dual Attention Net-works for Multimodal Reasoning and Matching. Proceedings of the 2017 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 21-26 July 2017, 2156-2164. [Google Scholar] [CrossRef] 
[11] Yu, Z., Yu, J., Fan, J. and Tao, D. (2017) Multi-Modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering. Proceedings of the 2017 IEEE Internation-al Conference on Computer Vision, Venice, 22-29 October 2017, 1839-1848. [Google Scholar] [CrossRef] 
[12] Cadene, R., Ben-Younes, H., Cord, M. and Thome, N. (2019) MUREL: Multimodal Relational Reasoning for Visual Question Answering. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, 15-20 June 2019, 1989-1998, [Google Scholar] [CrossRef] 
[13] Yu, Z., Yu, J., Cui, Y., et al. (2019) Deep Modular Co-Attention Networks for Visual Question Answering. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, 15-20 June 2019, 6281-6290. [Google Scholar] [CrossRef] 
[14] Chen, C., Han, D. and Wang, J. (2020) Multimodal Encod-er-Decoder Attention Networks for Visual Question Answering. IEEE Access, 8, 35662-35671. [Google Scholar] [CrossRef] 
[15] Guo, W., Zhang, Y. and Yang, J., et al. (2021) Re-Attention for Visual Question Answering. IEEE Transactions on Image Processing, 30, 6730-6743. [Google Scholar] [CrossRef] 
[16] Liu, Y., Zhang, X., Zhang, Q., et al. (2021) Dual Self-Attention with Co-Attention Networks for Visual Question Answering. Pattern Recognition, 117, Article ID: 107956. [Google Scholar] [CrossRef] 
[17] Sharma, H. and Jalal, A.S. (2022) An Improved Attention and Hybrid Optimization Technique for Visual Question Answering. Neural Processing Letters, 54, 709-730. [Google Scholar] [CrossRef] 
[18] Cornia, M., Stefanini, M., Baraldi, L., et al. (2020) Meshed-Memory Transformer for Image Captioning. Proceedings of the 2020 IEEE/CVF Conference on Computer Vi-sion and Pattern Recognition, Seattle, 13-19 June 2020, 10578-10587. [Google Scholar] [CrossRef] 
[19] Pennington, J., Socher, R. and Manning, C.D. (2014) Glove: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, 25-29 October 2014, 1532-1543. [Google Scholar] [CrossRef] 
[20] Goyal, Y., et al. (2019) Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. International Journal of Computer Vision, 127, 398-414. [Google Scholar] [CrossRef] 
[21] Bai, Y., Fu, J., Zhao, T. and Mei, T. (2018) Deep Attention Neu-ral Tensor Network for Visual Question Answering. In: Ferrari, V., Hebert, M., Sminchisescu, C. and Weiss, Y., Eds., ECCV 2018: Computer Vision—ECCV 2018, Lecture Notes in Computer Science, Vol. 11216, Springer, Cham, 21-37. [Google Scholar] [CrossRef] 
[22] Ben-Younes, H., Cadene, R., Cord, M., et al. (2017) Mutan: Multimodal Tucker Fusion for Visual Question Answering. Proceedings of the 2017 IEEE International Conference on Computer Vision, Venice, 22-29 October 2017, 2612-2620. [Google Scholar] [CrossRef] 
[23] Yang, Z., Qin, Z., Yu, J. and Wan, T. (2020) Prior Visual Relationship Reasoning for Visual Question Answering. Proceedings of 2020 IEEE International Conference on Image Processing (ICIP), Abu Dhabi, 25-28 October 2020, 1411-1415. [Google Scholar] [CrossRef] 
[24] Kim, J.H., Jun, J. and Zhang, B.T. (2018) Bilinear Attention Networks. Proceedings of 32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, 3-8 De-cember 2018, 31.