基于双水平路由注意力机制的时序Transformer临近预报研究
A Study of Temporal Transformer Nowcast Based on Bilevel Routing Attention Mechanism
DOI: 10.12677/csa.2024.1411230, PDF,   
作者: 卢 姁, 王也英, 李一玲, 董先飞：中国人民解放军32021部队，北京；初奕琦, 王 曦, 孙国栋：北京无线电测量研究所，北京
关键词: 临近预报；深度学习；双水平路由注意力；视觉Transformer；Nowcast； Deep Learning； Bilevel Routing Attention； Vision Transformer
摘要: 0~2小时的临近预报是预报强对流天气的重要手段，准确的临近预报是气象防灾的重要屏障。临近预报是气象预报的一个重要业务。本文提出了一个基于时序Transformer的全局特征提取的Encoder-Forecasting临近预报模型结构。该模型的Transformer利用双级路由注意力机制定位特征中的几个最相关的键值对来提高计算效率。模型同时采用MaxPool和AvgPool提取目标对象的局部特征，并将局部特征和全局特征进行有效融合，进而充分提取目标对象的特征用于临近预报。提出的模型与几个模型在一个公共的临近预报数据集上进行实验验证。实验结果充分证明了我们提出的模型的有效性和准确性。
Abstract: The nowcast of 0~2 hour is an important toolth for forecasting severe convective weather. Accurate nowcast is an important barrier to prevent from weather disaster. Therefore, nowcast is a very important field in the meteorological society. This paper proposes a nowcast model which based on Encoder-Forecasting architecture and temporal Transformer. The transformer utilizes bilevel routing attention to improve the computational efficiency. At the same time, the proposed model also use the operations of Maxpool and AvgPool to extract the local features. The global features and local features are effectively fused to represent the objects’ features. The comprehensive experiments based on the proposed model and four state-of-art models are conducted on a public datasets. The experimental results effectively show the correctness and effectiveness of the proposed model.
文章引用：卢姁, 王也英, 李一玲, 董先飞, 初奕琦, 王曦, 孙国栋. 基于双水平路由注意力机制的时序Transformer临近预报研究[J]. 计算机科学与应用, 2024, 14(11): 208-217. https://doi.org/10.12677/csa.2024.1411230

参考文献

[1] Han, L., Zhang, J., Chen, H., Zhang, W. and Yao, S. (2022) Toward the Predictability of a Radar-Based Nowcasting System for Different Precipitation Systems. IEEE Geoscience and Remote Sensing Letters, 19, 1-5. [Google Scholar] [CrossRef] 
[2] Reyniers, M. (2008) Quantitative Precipitation Forecasts Based on Radar Observations: Principles, Algorithms and Operational Systems. Institut Royal Météorologique de Belgique.
[3] Sun, J., Xue, M., Wilson, J.W., Zawadzki, I., Ballard, S.P., Onvlee-Hooimeyer, J., et al. (2014) Use of NWP for Nowcasting Convective Precipitation: Recent Progress and Challenges. Bulletin of the American Meteorological Society, 95, 409-426. [Google Scholar] [CrossRef] 
[4] Cheung, P. and Yeung, H.Y. (2012) Application of Optical-flow Technique to Significant Convection Nowcast for Terminal Areas in Hong Kong. The 3rd WMO International Symposium on Nowcasting and Very Short Range Forecasting (WSN12), Rio de Janeiro, 6-10 August 2012, 6-10.
[5] Rinehart, R.E. (1981) A Pattern Recognition Technique for Use with Conventional Weather Radar to Determine Internal Storm Motions. Atmos Technologies, 13, 119-134.
[6] 曹春燕, 陈元昭, 刘东华. 光流法及其在临近预报中的应用[J]. 气象学报, 2015, 73(3): 471-480.
[7] LeCun, Y., Bengio, Y. and Hinton, G. (2015) Deep learning. Nature, 521, 436-444. [Google Scholar] [CrossRef] [PubMed]
[8] Shi, X., Chen, Z. and Wang, H. (2015) Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. Advances in Neural Information Processing Systems 28 (NIPS 2015), Montreal, 7-12 December 2015, 802-810.
https://proceedings.neurips.cc/paper/2015/hash/07563a3fe3bbe7e3ba84431ad9d055af-Abstract.html
[9] Shi, X., Gao, Z. and Lausen, L. (2017) Deep Learning for Precipitation Nowcasting: A Benchmark and a New Model. Advances in Neural Information Processing Systems 30 (NIPS 2017), California, 4-9 December 2017, 5617-5627.
https://proceedings.neurips.cc/paper/2017/hash/a6db4ed04f1621a119799fd3d7545d3d-Abstract.html
[10] Wang, Y., Gao, Z., Long, M., et al. (2018) PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning. arXiv: 1804.06300.
[11] Trebing, K., Staǹczyk, T. and Mehrkanoon, S. (2021) SmaAT-UNet: Precipitation Nowcasting Using a Small Attention-UNet Architecture. Pattern Recognition Letters, 145, 178-186. [Google Scholar] [CrossRef] 
[12] Wang, Y., Wu, H., Zhang, J., Gao, Z., Wang, J., Yu, P.S., et al. (2023) PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45, 2208-2225. [Google Scholar] [CrossRef] [PubMed]
[13] Ravuri, S., Lenc, K., Willson, M., Kangin, D., Lam, R., Mirowski, P., et al. (2021) Skilful Precipitation Nowcasting Using Deep Generative Models of Radar. Nature, 597, 672-677. [Google Scholar] [CrossRef] [PubMed]
[14] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N. and Polosukhin, I. (2017) Attention Is All You Need. Advances in Neural Information Processing Systems, 30, 6000-6010.
[15] Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2021) An Image Is Worth 16 x 16 Words: Transformers for Image Recognition at Scale. Proceedings of the International Conference on Learning Representations, Vienna, 4 May 2021, 11929.
[16] Bai, C., Sun, F., Zhang, J., Song, Y. and Chen, S. (2022) Rainformer: Features Extraction Balanced Network for Radar-Based Precipitation Nowcasting. IEEE Geoscience and Remote Sensing Letters, 19, 1-5. [Google Scholar] [CrossRef] 
[17] Li, D., Deng, K., Zhang, D., Liu, Y., Leng, H., Yin, F., et al. (2023) LPT-QPN: A Lightweight Physics-Informed Transformer for Quantitative Precipitation Nowcasting. IEEE Transactions on Geoscience and Remote Sensing, 61, 1-19. [Google Scholar] [CrossRef] 
[18] Chen, S., Shu, T., Zhao, H., Zhong, G. and Chen, X. (2023) TempEE: Temporal-Spatial Parallel Transformer for Radar Echo Extrapolation Beyond Autoregression. IEEE Transactions on Geoscience and Remote Sensing, 61, 1-14. [Google Scholar] [CrossRef] 
[19] Zhu, L., Wang, X., Ke, Z., Zhang, W. and Lau, R. (2023) BiFormer: Vision Transformer with Bi-Level Routing Attention. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, 17-24 June 2023, 10323-10333. [Google Scholar] [CrossRef] 
[20] Veillette, M., Samsi, S. and Mattioli, C. (2020) SEVIR: A Storm Event Imagery Dataset for Deep Learning Applications in Radar and Satellite Meteorology. Advances in Neural Information Processing Systems, 33, 22009-22019.
[21] Gao, Z., Shi, X., Wang, H., et al. (2023) Earthformer: Exploring Space-Time Transformers for Earth System Forecasting. Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, 28 November-9 December 2022, 2539-2540.
[22] Yu, D., Li, X., Ye, Y., Zhang, B., Luo, C., Dai, K., et al. (2024) DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcasting. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, 16-22 June 2024, 27758-27767. [Google Scholar] [CrossRef]