基于门控姿态融合与时序记忆加权的时空动作检测方法
Spatio-Temporal Action Detection Based on Gated Pose Fusion and Temporal Memory Weighting
摘要: 时空动作检测需要同时定位视频中人物动作的空间位置并判断动作类别。Holistic Interaction Transformer Network (HIT)通过RGB分支、姿态分支和时序记忆信息进行多模态交互建模,但原始结构对姿态分支可靠性和不同时序记忆特征相关性的区分仍不充分。针对该问题,本文在HIT基础上提出一种轻量改进方法:一方面设计门控姿态融合模块,根据RGB ROI特征与姿态交互特征生成门控权重,自适应调节姿态特征对动作表征的贡献;另一方面设计时序距离感知记忆加权模块,根据记忆帧与当前帧的时间距离对memory feature进行衰减加权,以增强近邻上下文并降低远距离特征干扰。在JHMDB-21数据集上的实验结果表明,最终组合模型在Video mAP@0.5和Video mAP@0.2上分别达到88.02%和89.55%,相较自训练基线分别提升1.88和2.15个百分点。实验结果说明,所提轻量模块能够提升HIT在视频级时空动作检测中的稳定性。
Abstract: Spatio-temporal action detection requires recognizing action categories while localizing actors in video frames. The Holistic Interaction Transformer Network (HIT) provides a multimodal interaction framework with RGB features, pose features and temporal memory information, but its original structure still lacks explicit reliability modeling for pose fusion and temporal memory utilization. To address this problem, this paper proposes a lightweight improvement based on HIT. First, a gated pose fusion module is introduced to generate adaptive weights from RGB ROI features and pose interaction features, thereby controlling the contribution of the pose branch. Second, a temporal distance-aware memory weighting module is designed to reweight memory features according to their temporal distance from the current frame, to enhance nearby context and suppress the interference from distant features. Experiments on JHMDB-21 show that the final model achieves 88.02% Video mAP@0.5 and 89.55% Video mAP@0.2, improving the self-trained baseline by 1.88 and 2.15 percentage points, respectively. The results indicate that the proposed lightweight modules improve the video-level stability of HIT for spatio-temporal action detection.
文章引用:朱强. 基于门控姿态融合与时序记忆加权的时空动作检测方法[J]. 计算机科学与应用, 2026, 16(11): 216-227. https://doi.org/10.12677/csa.2026.1611338

参考文献

[1] Kalogeiton, V., Weinzaepfel, P., Ferrari, V. and Schmid, C. (2017) Action Tubelet Detector for Spatio-Temporal Action Localization. 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 22-29 October 2017, 4415-4423.
https://doi.org/10.1109/iccv.2017.472
[2] Feichtenhofer, C., Fan, H., Malik, J. and He, K. (2019) SlowFast Networks for Video Recognition. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, 27 October-2 November 2019, 6201-6210.
https://doi.org/10.1109/iccv.2019.00630
[3] Faure, G.J., Chen, M.H. and Lai, S.H. (2023) Holistic Interaction Transformer Network for Action Detection. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, 2-7 January 2023, 3329-3339.
https://doi.org/10.1109/wacv56688.2023.00334
[4] Jhuang, H., Gall, J., Zuffi, S., Schmid, C. and Black, M.J. (2013) Towards Understanding Action Recognition. 2013 IEEE International Conference on Computer Vision, Sydney, 1-8 December 2013, 3192-3199.
https://doi.org/10.1109/iccv.2013.396
[5] Ren, S., He, K., Girshick, R. and Sun, J. (2015) Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems 28, Montreal, 7-12 December 2015, 1137-1149.
[6] He, K., Gkioxari, G., Dollar, P. and Girshick, R. (2017) Mask R-CNN. 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 22-29 October 2017, 2980-2988.
https://doi.org/10.1109/iccv.2017.322
[7] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L. and Polosukhin, I. (2017) Attention Is All You Need. Advances in Neural Information Processing Systems 30, Long Beach, 4-9 December 2017, 5998-6008.
[8] Arevalo, J., Solorio, T., Montes, M. and Gonzalez, F.A. (2017) Gated Multimodal Units for Information Fusion. 2017 ICLR Workshop, Toulon, 24-26 April 2017, 1-12.
[9] Hu, J., Shen, L. and Sun, G. (2018) Squeeze-and-Excitation Networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-23 June 2018, 7132-7141.
https://doi.org/10.1109/cvpr.2018.00745
[10] Wang, X., Girshick, R., Gupta, A. and He, K. (2018) Non-Local Neural Networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-23 June 2018, 7794-7803.
https://doi.org/10.1109/cvpr.2018.00813