基于动态卷积的高光谱图像融合网络
Hyperspectral Image Fusion Network Based on Dynamic Convolution
DOI: 10.12677/mos.2024.133300, PDF,   
作者: 冯佳琪：上海理工大学光电信息与计算机工程学院，上海
关键词: 高光谱图像融合；深度学习；动态卷积；Hyperspectral Image Fusion； Deep Learning； Dynamic Convolution
摘要: 基于深度学习的高光谱(Hyperspectral Image, HSI)和多光谱图像(Multispectral Image, MSI)融合技术已经被广泛研究，以提高高光谱图像的分辨率。但大多数方法在融合时不能充分提取高光谱和多光谱图像的有效信息。针对这个问题，本文提出了一种基于动态卷积的高光谱图像融合网络DHIF，主要通过三个分支进行图像的特征提取和融合重构。首先，在特征提取阶段，引入动态卷积来提取高光谱和多光谱图像的串联信息，然后针对其他两个图像分支，设计高光谱动态卷积子网络HDCN和多光谱动态卷积子网络MDCN，分别提取HSI和MSI的光谱与空间信息。此外，提取到的信息一方面用于HR-HSI的重构，另一方面作为损失函数的一部分来约束网络的训练。本文在三个高光谱数据集Pavia University (PU)、 Pavia Centre (PC)和Botswana上实现了DHIF，并和其他九种目前较好的融合算法进行比较，证明本文提出的模型不论是在数量上还是质量上，都实现了最好的融合效果。
Abstract: In order to improve the resolution of hyperspectral image (HSI), many hyperspectral and multispectral image (MSI) fusion methods based on deep learning have been researched widely. However, when fusing, most of them don’t make full use of the important information of HSI and MSI. To tackle this issue, we propose a hyperspectral and multispectral image fusion network based on dynamic convolution DHIF, where the features of HSI and MSI are extracted from three branches and then used to reconstruct HR-HSI. First, in the stage of feature extraction, we introduce dynamic convolution to capture the series information of HSI and MSI, and then design hyperspectral dynamic convolutional subnetwork HDCN and multispectral dynamic convolutional subnetwork MDCN to utilize the spectral and spatial features of two both, respectively. Third, the extracted features from three branches are not only inputted into the stage of image reconstruction but also added to the loss function to constrain the network training. Extensive experiments on three datasets of Pavia University (PU), Pavia Centre (PC), and Botswana demonstrate that the proposed DHIF surpasses other nine state-of-the-art methods and performs best both in quantitative and qualitative terms.
文章引用：冯佳琪. 基于动态卷积的高光谱图像融合网络[J]. 建模与仿真, 2024, 13(3): 3292-3305. https://doi.org/10.12677/mos.2024.133300

参考文献

[1] Mou, L., Ghamisi, P. and Zhu, X.X. (2017) Deep Recurrent Neural Networks for Hyperspectral Image Classification. IEEE Transactions on Geoscience and Remote Sensing, 55, 3639-3655. [Google Scholar] [CrossRef] 
[2] Wang, Q., He, X. and Li, X. (2018) Locality and Structure Regularized Low Rank Representation for Hyperspectral Image Classification. IEEE Transactions on Geoscience and Remote Sensing, 57, 911-923. [Google Scholar] [CrossRef] 
[3] Xie, Z., Li, Y., Niu, J., Yu, X. and Shi, L. (2019) Hyperspectral Face Recognition Using Block Based Convolution Neural Network and AdaBoost Band Selection. 2019 6th International Conference on Systems and Informatics (ICSAI), Shanghai, 2-4 November 2019, 1270-1274.
[4] Yan, H., Zhang, Y., Wei, W., et al. (2016) Salient Object Detection in Hyperspectral Imagery Using Spectral Gradient Contrast. 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Beijing, 10-15 July 2016, 1560-1563. [Google Scholar] [CrossRef] 
[5] Zhou, J., Kwan, C., Ayhan, B. and Eismann, M.T. (2016) A Novel Cluster Kernel RX Algorithm for Anomaly and Change Detection Using Hyperspectral Images. IEEE Transactions on Geoscience and Remote Sensing, 54, 6497-6504. [Google Scholar] [CrossRef] 
[6] Li, S., Zhang, K., Hao, Q., Duan, P. and Kang, X. (2018) Hyperspectral Anomaly Detection with Multiscale Attribute and Edge-Preserving Filters. IEEE Geoscience and Remote Sensing Letters, 15, 1605-1609. [Google Scholar] [CrossRef] 
[7] Hu, J.-F., Huang, T.-Z., Deng, L.-J., Jiang, T.-X., Vivone, G. and Chanussot, J. (2021) Hyperspectral Image Super-Resolution via Deep Spatiospectral Attention Convolutional Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 33, 7251-7265. [Google Scholar] [CrossRef] 
[8] Dian, R., Li, S. and Kang, X. (2020) Regularizing Hyperspectral and Multispectral Image Fusion by CNN Denoiser. IEEE Transactions on Neural Networks and Learning Systems, 32, 1124-1135. [Google Scholar] [CrossRef] 
[9] Dong, W., Fu, F., Shi, G., Cao, X., Wu, J., Li, G. and Li, X. (2016) Hyperspectral Image Super-Resolution via Non-Negative Structured Sparse Representation. IEEE Transactions on Image Processing, 25, 2337-2352. [Google Scholar] [CrossRef] 
[10] Lin, C.-H., Ma, F., Chi, C.-Y. and Hsieh, C.-H. (2017) A Convex Optimization-Based Coupled Nonnegative Matrix Factorization Algorithm for Hyperspectral and Multispectral Data Fusion. IEEE Transactions on Geoscience and Remote Sensing, 56, 1652-1667. [Google Scholar] [CrossRef] 
[11] Han, X.-H., Shi, B. and Zheng, Y. (2018) Self-Similarity Constrained Sparse Representation for Hyperspectral Image Super-Resolution. IEEE Transactions on Image Processing, 27, 5625-5637. [Google Scholar] [CrossRef] 
[12] Wang, X., Cheng, C., Liu, S., Song, R., Wang, X. and Feng, L. (2023) SS-INR: Spatial-Spectral Implicit Neural Representation Network for Hyperspectral and Multispectral Image Fusion. IEEE Transactions on Geoscience and Remote Sensing, 61, 1-14. [Google Scholar] [CrossRef] 
[13] Chen, Y., Dai, X., Liu, M., et al. (2020) Dynamic Convolution: Attention over Convolution Kernels. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, 13-19 June 2020, 11027-11036. [Google Scholar] [CrossRef] 
[14] Zhao, Z. and Dong, M. (2023) Channel-Spatial Dynamic Convolution: An Exquisite Omni-Dimensional Dynamic Convolution. 2023 8th International Conference on Intelligent Computing and Signal Processing (ICSP), Xi’an, 21-23 April 2023, 1707-1711. [Google Scholar] [CrossRef] 
[15] Gross, S. and Wilber, M. (2016) Training and Investigating Residual Nets. Facebook AI Research.
[16] Yuhas, R.H., Goetz, A.F.H. and Boardman, J.W. (1992) Discrimination among Semi-Arid Landscape Endmembers Using the Spectral Angle Mapper (SAM) Algorithm. Summaries of the Third Annual JPL Airborne Geoscience Workshop, Vol. 1, Pasadena, 1-5 June 1992, 147-149.
[17] Wang, Z., Bovik, A.C., Sheikh, H.R. and Simoncelli, E.P. (2004) Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing, 13, 600-612. [Google Scholar] [CrossRef] 
[18] Wald, L. (2000) Quality of High Resolution Synthesised Images: Is There a Simple Criterion? Proceedings of the Third Conference “Fusion of Earth Data: Merging Point Measurements, Raster Maps and Remotely Sensed Images”, Sophia Antipolis, 26-28 January 2000, 99-103.
[19] Han, X.-H., Zheng, Y. and Chen, Y.-W. (2019) Multi-Level and Multi-Scale Spatial and Spectral Fusion CNN for Hyperspectral Image Super-Resolution. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, 27-28 October 2019, 4330-4339.
[20] Xu, T., Huang, T.-Z., Deng, L.-J. and Yokoya, N. (2022) An Iterative Regularization Method Based on Tensor Subspace Representation for Hyperspectral Image Super-Resolution. IEEE Transactions on Geoscience and Remote Sensing, 60, 1-16. [Google Scholar] [CrossRef] 
[21] Liu, X., Liu, Q. and Wang, Y. (2020) Remote Sensing Image Fusion Based on Two-Stream Fusion Network. Information Fusion, 55, 1-15. [Google Scholar] [CrossRef] 
[22] He, L., Zhu, J., Li, J., Plaza, A., Chanussot, J. and Li, B. (2019) HyperPNN: Hyperspectral Pansharpening via Spectrally Predictive Convolutional Neural Networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12, 3092-3100. [Google Scholar] [CrossRef] 
[23] Zhang, X., Huang, W., Wang, Q. and Li, X. (2020) SSR-NET: Spatial-Spectral Reconstruction Network for Hyperspectral and Multispectral Image Fusion. IEEE Transactions on Geoscience and Remote Sensing, 59, 5953-5965. [Google Scholar] [CrossRef] 
[24] Yuan, Q., Wei, Y., Meng, X., Shen, H. and Zhang, L. (2018) A Multiscale and Multidepth Convolutional Neural Network for Remote Sensing Imagery Pan-Sharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11, 978-989. [Google Scholar] [CrossRef] 
[25] Zheng, Y., Li, J., Li, Y., Guo, J., Wu, X. and Chanussot, J. (2020) Hyperspectral Pansharpening Using Deep Prior and Dual Attention Residual Network. IEEE Transactions on Geoscience and Remote Sensing, 58, 8059-8076. [Google Scholar] [CrossRef] 
[26] Qu, J., Hou, S., Dong, W., Xiao, S., Du, Q. and Li, Y. (2021) A Dual-Branch Detail Extraction Network for Hyperspectral Pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 60, 1-13. [Google Scholar] [CrossRef] 
[27] Han, X.H., Shi, B. and Zheng, Y. ((2018) SSF-CNN: Spatial and Spectral Fusion with CNN for Hyperspectral Image Super-Resolution. 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, 7-10 October 2018, 2506-2510. [Google Scholar] [CrossRef] 
[28] Min, Z., Wang, Y. and Jia, S. (2021) Multiscale Spatial-Spectral Joint Feature Learning for Multispectral and Hyperspectral Image Fusion. 2021 IEEE 23rd International Conference on High Performance Computing & Communications; 7th International Conference on Data Science & Systems; 19th International Conference on Smart City; 7th International Conference on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/ DependSys), Haikou, 20-22 December 2021, 1265-1270. [Google Scholar] [CrossRef] 

为你推荐