深度学习和APAP模型结合的大视差图像拼接算法
Large Parallax Image Mosaic Algorithm Based on Deep Learning and APAP Model
DOI: 10.12677/JISP.2023.122011, PDF,    科研立项经费支持
作者: 熊 禹：广西科技大学自动化学院，广西 柳州；东风柳州汽车有限公司，广西 柳州；尹希庆*, 吴志运：广西科技大学自动化学院，广西 柳州
关键词: 图像拼接；深度学习；APAP模型；SuperPoint网络；SuperGlue网络；Image Mosaic； Deep Learning； APAP Model； SuperPoint Network； SuperGlue Network
摘要: 图像间位置偏差较大时，图像拼接容易出现错位、重影问题。提出一种深度学习和APAP模型结合的大视差图像拼接算法，该算法使用基于学习的SuperPoint网络同时提取图像特征点和描述符，采用SuperGlue网络对特征点进行筛选和最优匹配，最后通过APAP模型求取局部投影变换完成拼接。实际结果显示，在Mikolajczyk数据集上，基于学习的SuperPoint和SuperGlue网络在特征点提取和匹配方面相比传统算法重复率提升20%左右，准确率达到99%，鲁棒性更强，准确率更高。最终拼接图相比传统算法图像质量评价指标NIQE降低6.5%左右，基本消除错位、重影问题，更符合视觉效果。
Abstract: When the position deviation between images is large, image Mosaic is easy to appear dislocation and double image problems. A large parallax image stitching algorithm combining deep learning and APAP model is proposed. The algorithm uses learn-based SuperPoint network to simultaneously extract image feature points and descriptors, and uses SuperGlue network to screen and optimize the feature points. Finally, APAP model is used to obtain local projection transform to complete the stitching. The actual results show that on the Mikolajczyk dataset, the repetition rate of the SuperPoint and SuperGlue networks based on learning is improved by about 20% compared with the traditional algorithm in feature point extraction and matching, and the accuracy rate is up to 99%, with stronger robustness and higher accuracy. Compared with the image quality evaluation index NIQE of the traditional algorithm, the final Mosaic image is about 6.5% lower, basically eliminating the dislocation and double image problems, and is more in line with the visual effect.
文章引用：熊禹, 尹希庆, 吴志运. 深度学习和APAP模型结合的大视差图像拼接算法[J]. 图像与信号处理, 2023, 12(2): 104-115. https://doi.org/10.12677/JISP.2023.122011

参考文献

[1] Tafti, A.P., Baghaie, A., Kirkpatrick, A.B., et al. (2018) A Comparative Study on the Application of SIFT, SURF, BRIEF and ORB for 3D Surface Reconstruction of Electron Microscopy Images. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 6, 17-30.
[Google Scholar] [CrossRef] 
[2] Anderson, R., Gallup, D., Barron, J.T., et al. (2016) Jump: Virtual Reality Video. ACM Transactions on Graphics, 35, Article No. 198.
[Google Scholar] [CrossRef] 
[3] Lin, H.S., Chang, C.C., Chang, H.Y., et al. (2018) A Low-Cost Portable Polycamera for Stereoscopic 360 Imaging. IEEE Transactions on Circuits and Systems for Video Technology, 29, 915-929.
[Google Scholar] [CrossRef] 
[4] 李加亮, 蒋品群, 夏海英. 基于网格变形和余弦函数权重的图像拼接方法[J]. 广西师范大学学报(自然科学版), 2020, 38(4): 42-53.
[Google Scholar] [CrossRef] 
[5] 薛佳乐, 赵萌, 张哲, 程徐, 陈胜勇. 针对大视差图像拼接的显性子平面配准[J]. 中国图象图形学报, 2018, 23(3): 323-332.
[6] 何川, 周军. 具有直线结构保护的网格化图像拼接[J]. 中国图象图形学报, 2018, 23(7): 973-983.
[7] Lowe, D.G. (2004) Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision, 60, 91-110.
[Google Scholar] [CrossRef] 
[8] Bay, H., Ess, A., Tuytelaars, T., et al. (2008) Speeded-Up Robust Features (SURF). Computer Vision and Image Understanding, 110, 346-359.
[Google Scholar] [CrossRef] 
[9] Rublee, E., Rabaud, V., Konolige, K., et al. (2011) ORB: An Efficient Alternative to SIFT or SURF. 2011 International Conference on Computer Vision, Barcelona, 6-13 November 2011, 2564-2571.
[Google Scholar] [CrossRef] 
[10] Gao, J., Kim, S.J. and Brown, M.S. (2011) Constructing Image Panoramas Using Dual-Homography Warping. CVPR 2011, Colorado Springs, CO, 20-25 June 2011, 49-56.
[Google Scholar] [CrossRef] 
[11] 许越, 徐之海, 冯华君, 李奇, 陈跃庭, 徐毅, 赵洪波. 双场景类型遥感图像的配准拼接优化[J]. 浙江大学学报(工学版), 2019, 53(1): 107-114.
[12] Zaragoza, J., Chin, T.J., Brown, M.S., et al. (2013) As-Projective-As-Possible Image Stitching with Moving DLT. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, 23-28 June 2013, 2339-2346.
[Google Scholar] [CrossRef] 
[13] Chang, C.H., Sato, Y. and Chuang, Y.Y. (2014) Shape-Preserving Half-Projective Warps for Image Stitching. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, 23-28 June 2014, 3254-3261.
[Google Scholar] [CrossRef] 
[14] Lin, C.C., Pankanti, S.U., Natesan, R.K, et al. (2015) Adaptive As-Natural-As-Possible Image Stitching. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, 7-12 June 2015, 1155-1163.
[Google Scholar] [CrossRef] 
[15] 屈玳辉, 谢益武. 基于APAP模型的大视差图像拼接算法[J]. 测控技术, 2021, 40(1): 34-39.
[Google Scholar] [CrossRef] 
[16] DeTone, D., Malisiewicz, T. and Rabinovich, A. (2018) Superpoint: Self-Supervised Interest Point Detection and Description. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, 18-22 June 2018, 337-33712.
[Google Scholar] [CrossRef] 
[17] Sarlin, P.E, DeTone, D., Malisiewicz, T., et al. (2020) Superglue: Learning Feature Matching with Graph Neural Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, 13-19 June 2020, 4938-4947.
[Google Scholar] [CrossRef] 
[18] 马东岭, 吴鼎辉, 毛力波, 石壮. 基于最优接缝线的加权融合影像拼接算法[J]. 测绘工程, 2020, 29(6): 1-7.
[Google Scholar] [CrossRef] 
[19] Simonyan, K. and Zisserman, A. (2014) Very Deep Convolutional Networks for Large-Scale Image Recognition. https://arxiv.org/abs/1409.1556
[20] Schonberger, J.L. and Frahm, J.M. (2016) Structure-from-Motion Revisited. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, 27-30 June 2016, 4104-4113.
[Google Scholar] [CrossRef] 
[21] Fischler, M.A. and Bolles, R.C. (1981) Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Communications of the ACM, 24, 381-395.
[Google Scholar] [CrossRef] 
[22] Mikolajczyk, K. and Schmid, C. (2005) A Performance Evaluation of Local Descriptors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 1615-1630.
[Google Scholar] [CrossRef] 
[23] Mittal, A., Soundararajan, R. and Bovik, A.C. (2013) Making a “Completely Blind” Image Quality Analyzer. IEEE Signal Processing Letters, 20, 209-212.
[Google Scholar] [CrossRef]