利用文本特征增强与注意力机制提高图像问答准确率

Improve Image Question and Answer Accuracy by Using Text Feature Enhancement and Attention Mechanism

摘要: 图像问答是深度学习在计算机视觉领域成功应用的主要方向之一，在人工智能、自然语言处理、图像识别等方面有着广泛应用。图像问答的准确率不仅与图像问答系统中特征融合模块的设计有关，而且与图像特征与问题特征语义层次匹配程度有关。本文首先将图像的文本特征和视觉特征融合后作为图像增强特征，之后对问题提取文本特征，再加入注意力机制，将图像增强特征与问题文本特征进行特征融合，对融合特征做出答案预测。实验结果表明，本文方法可以解决图像特征与文本特征层次不匹配的问题，提高图像问答系统的准确率。

Abstract: Image question and answer is one of the main directions for the successful application of deep learning in the field of computer vision. It has been widely used in artificial intelligence, natural language processing, image recognition and so on. The accuracy of the image question and answer is not only related to the design of the feature fusion module in the image question answering system, but also related to the degree of matching between the image feature and the semantic level of the question feature. In this paper, the text features and visual features of the image are first combined as the enhanced features of the image. Then, the text features are extracted from the question, and then the attention mechanism is added. The enhanced features of the image and the text features of the question are merged, and make answer prediction for fusion features. The experimental results show that the proposed method can solve the problem of mismatch between image features and text features, and improve the accuracy of the image question answering system.