基于深度学习的语义场景图像检索

Deep Learning Based Semantic Scene Image Retrieval

摘要: 随着互联网图像等多媒体内容的爆炸式增长，在线Web图像的语义场景检索问题引起了学者们的研究兴趣。传统的研究工作聚焦在基于单概念的图像检索上，未能很好检索含有复杂语义场景的图像。为解决语义场景Web图像检索问题，我们提出了一种基于多模态深度学习的语义场景图像检索方法(SSIR)。首先，使用一个多模态CNN训练网络作为概念分类器；其次，通过计算语义概念之间的依赖关系来精炼概念的语义分数，以进一步增强分类器的场景识别能力；最后，为提升对稀疏场景概念的检索性能，应用梯度下降算法来补偿在真实应用中不平衡图像集上语义概念的频率差。在MIR Flickr 2011标准图像数据集上对比了其他传统方法，结果表明我们的语义场景检索方法性能更优。

Abstract: With the explosive growth of multimedia objects such as Web images over the Internet, online semantic scene image retrieval has been receiving increasing research interest. Conventional studies focus on single-concept-based image retrieval and cannot effectively retrieve semantic scene images including multiple concepts that describe characteristic semantic scene. To tackle this issue, i.e., semantic scene Web image retrieval, we propose a novel approach called multi-modal deep learning based Semantic Scene Image Retrieval (SSIR) in this paper. In particular, we first train a multi-modal Convolutional Neural Network (CNN) as a concept classifier for images and texts. Second, semantic interdependencies of the subconcepts included in the images are utilized to refine the predicted semantic scores in order to enhance holistic scene recognition. Finally, to improve the performance of retrieving rare scene concepts, a gradient descent algorithm is used for compensating the varying frequencies of concepts derived from imbalanced image datasets. The results of our experiments on MIR Flickr 2011 have shown that our proposed approach performs favorably compared with several traditional methods.