基于深度学习的点云上采样算法研究
Research on Point Cloud Upsampling Algorithms Based on Deep Learning
DOI: 10.12677/JISP.2023.121003, PDF,  被引量    国家自然科学基金支持
作者: 王皓辰, 张长伦*：北京建筑大学理学院，北京；黎铭亮：北京建筑大学，北京
关键词: 深度学习；点云上采样；Deep Learning； Point Cloud Upsampling
摘要: 点云上采样能够提高点云分辨率并保持点云的特征，近年来越来越受到人们的重视。基于深度学习的点云上采样算法相较于基于优化的算法，能够更有效地学习点云的特征和结构，且对数据的先验要求不高，取得了先进的上采样效果。因此基于深度学习的点云上采样是当前许多学者主要研究的方向之一。本文综述了基于深度学习的点云上采样算法，阐述了点云上采样的整体框架以及改进的策略，并介绍了点云上采样效果的评价指标以及常用的数据集，最后探讨了点云上采样的未来的几个极具潜力的发展方向。
Abstract: Point cloud upsampling improves the resolution of point cloud and maintains the feature of point cloud, which has attracted more and more attention in recent years. Compared with the optimization-based algorithms, the point cloud upsampling algorithms based on deep learning can more effectively learn the feature and structure of the point cloud and have low prior requirements for data, leading to the advanced effect of upsampling. Therefore, the point cloud upsampling based on deep learning is one of the main research directions of many scholars at present. In this paper, we summarize the point cloud upsampling algorithms based on deep learning and expound the holistic framework and improved strategies of point cloud upsampling. Then the evaluation metrics of point cloud upsampling effect and commonly used data sets are introduced. We finally discuss several potential development directions of point cloud upsampling in the future.
文章引用：王皓辰, 张长伦, 黎铭亮. 基于深度学习的点云上采样算法研究[J]. 图像与信号处理, 2023, 12(1): 21-31. https://doi.org/10.12677/JISP.2023.121003

参考文献

[1] Hoppe, H., DeRose, T., Duchamp, T., et al. (1992) Surface Reconstruction from Unorganized Points. Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), Chicago, 26-31 July 1992, 71-78.
[Google Scholar] [CrossRef] 
[2] Kazhdan, M.M. and Hoppe, H. (2013) Screened Poisson Surface Reconstruction. ACM Transactions on Graphics (TOG), 32, 29:1-29:13.
[Google Scholar] [CrossRef] 
[3] Newcombe, R.A., Izadi, S., Hilliges, O., et al. (2011) KinectFusion: Real-Time Dense Surface Mapping and Tracking. IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Basel, 26-29 October 2011, 127-136.
[Google Scholar] [CrossRef] 
[4] Riegler, G., Ulusoy, A.O., Bischof, H. and Geiger, A. (2017) Octnetfusion: Learning Depth Fusion from Data. IEEE International Conference on 3D Vision (3DV), Qingdao, 10-12 October 2017, 57-66.
[Google Scholar] [CrossRef] 
[5] Lang, A.H., Vora, S., Caesar, H., et al. (2019) Pointpillars: Fast Encoders for Object Detection from Point Clouds. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, 16-20 June 2019, 12697- 12705.
[Google Scholar] [CrossRef] 
[6] Wang, Y., Chao, W.-L., Garg, D., et al. (2019) Pseudolidar from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, 16-20 June 2019, 8445-8453.
[Google Scholar] [CrossRef] 
[7] Qi, C.R., Su, H., Mo, K. and Guibas, L.J. (2017) PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 21-26 July 2017, 77-85.
[8] Qi, C.R., Yi, L., Su, H. and Guibas, L.J. (2017) PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. International Conference on Neural Information Processing Systems (NIPS), Long Beach, 4-9 December 2017, 5099-5108.
[9] Yu, L.Q., Li, X.Z., et al. (2018) Pu-Net: Point Cloud Upsampling Network. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, 18-23 June 2018, 2790-2799.
[10] Yu, L., Li, X.Z., Fu, C.W., et al. (2018) Ec-net: An Edge-Aware Point Set Consolidation Network. ECCV 15th European Conference, Munich, 8-14 September 2018, 386-402.
[11] Qian, Y., Hou, J., Kwong, S. and He, Y. (2020) Pugeo-net: A Geometry-Centric Network for 3D Point Cloud Upsampling. 16th European Conference, Glasgow, 23-28 August 2020, 752-769.
[Google Scholar] [CrossRef] 
[12] Sharma, R., Schwandt, T., Kunert, C., Urban, S. and Broll, W. (2021) Point Cloud Upsampling and Normal Estimation Using Deep Learning for Robust Surface Reconstruction. Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021), Volume 5, 70-79.
[Google Scholar] [CrossRef] 
[13] Qian, G.C., Abualshour, A., Li, G.H., et al. (2021) Pu-gcn: Point Cloud Upsampling Using Graph Convolutional Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, 19-25 June 2021, 11683-11692.
[Google Scholar] [CrossRef] 
[14] Long, C., Zhang, W., Li, R., Wang, H., Dong, Z. and Yang, B. (2022) PC2-PU: Patch Correlation and Point Correlation for Effective Point Cloud Upsampling. Proceedings of the 30th ACM International Conference on Multimedia, Lisbon, 10-14 October 2022, 2191-2201.
[15] 钟帆, 柏正尧. 采用动态残差图卷积的3D点云超分辨率[J]. 浙江大学学报(工学版), 2022, 56(11): 2251-2259.
[16] Gu, F., Zhang, C.L., Wang, H.Y., et al. (2022) PU-WGCN: Point Cloud Upsampling Using Weighted Graph Convolutional Networks. Remote Sensing, 14, 5356.
[Google Scholar] [CrossRef] 
[17] Wang, Y.F., Wu, S.H., Huang, H., et al. (2019) Patch-Based Progressive 3d Point Set Upsampling. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, 16-20 June 2019, 5951-5960.
[Google Scholar] [CrossRef] 
[18] Li, R.H., Li, X.Z., Heng, P.-A. and Fu, C.-W. (2021) Point Cloud Upsampling via Disentangled Refinement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, 20-25 June 2021, 344-353.
[19] Du, H., Yan, X., Wang, J., Xie, D. and Pu, S. (2022) Point Cloud Upsampling via Cascaded Refinement Network.
[20] Li, R.H., Li, X.Z., Fu, C.W., et al. (2019) Pu-gan: A Point Cloud Upsampling Adversarial Network. IEEE International Conference on Computer Vision (ICCV), Seoul, 27 October-2 November 2019, 7202-7211.
[21] Wu, H.K., Zhang, J.G. and Huang, K.Q. (2020) Point Cloud Super Resolution with Adversarial Residual Graph Networks. British Machine Vision Conference (BMVC), Manchester, 7-11 September 2020, 256-267.
[22] Zhou, K., Dong, M. and Arslanturk, S. (2022) “Zero-Shot” Point Cloud Upsampling. 2022 IEEE International Conference on Multimedia and Expo (ICME), Taipei, 11-15 July 2022, 1-6.
[Google Scholar] [CrossRef] 
[23] Ye, S., Chen, D., Han, S., Wan, Z. and Liao, J. (2021) Meta-PU: An Arbitrary-Scale Upsampling Network for Point Cloud. IEEE Transactions on Visualization and Computer Graphics, 28, 3206-3218.
[Google Scholar] [CrossRef] 
[24] Feng, W., Li, J., Cai, H., Luo, X. and Zhang, J. (2022) Neural Points: Point Cloud Representation with Neural Fields for Arbitrary Upsampling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, 19-24 June 2022, 18633-18642.
[Google Scholar] [CrossRef] 
[25] Liu, X., Han, Z., Wen, X., et al. (2019) L2G Auto-Encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attention. Proceedings of the ACM International Conference on Multimedia, Nice, 21-25 October 2019, 989-997.
[Google Scholar] [CrossRef] 
[26] Liu, X., Liu, X., Liu, Y.S. and Han, Z. (2022) Spu-net: Self-Supervised Point Cloud Upsampling by Coarse-to-Fine Reconstruction with Self-Projection Optimization. IEEE Transactions on Image Processing, 31, 4213-4226.
[Google Scholar] [CrossRef] 
[27] Zhao, W., Liu, X., Zhong, Z., Jiang, J., Gao, W., Li, G. and Ji, X. (2022) Self-Supervised Arbitrary-Scale Point Clouds Upsampling via Implicit Neural Representation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, 19-24 June 2022, 1999-2007.
[Google Scholar] [CrossRef] 
[28] Qian, Y., Hou, J. and Kwong, S. (2021) Deep Magnification-Flexible Upsampling over 3D Point Clouds. IEEE Transactions on Image Processing, 30, 8354-8367.
[Google Scholar] [CrossRef] 
[29] Mao, A., Du, Z., Hou, J., Duan, Y., Liu, Y.J. and He, Y. (2022) PU-Flow: A Point Cloud Upsampling Network with Normalizing Flows. IEEE Transactions on Visualization and Computer Graphics, 1-14.
[Google Scholar] [CrossRef] 
[30] Nguyen, T., Pham, Q.H., Le, T., Pham, T., Ho, N. and Hua, B.S. (2021) Point-Set Distances for Learning Representations of 3D Point Clouds. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, 10-17 October 2021, 10478-10487.
[Google Scholar] [CrossRef] 
[31] Jesorsky, O., Klaus, J.K. and Robert, W.F. (2001) Robust Face Detection Using the Hausdorff Distance. International Conference on Audio- and Video-Based Biometric Person Authentication, Heidelberg, 6-8 June 2001, 90-95.
[Google Scholar] [CrossRef] 
[32] Fan, H.Q., Su, H. and Guibas, L.J. (2017) A Point Set Generation Network for 3D Object Reconstruction from a Single Image. CVPR 2017, Honolulu, 21-26 July 2017, 605-613.
[33] Sokolova, M., Japkowicz, N. and Szpakowicz, S. (2006) Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation. 19th Australian Joint Conference on Artificial Intelligence, Hobart, 4-8 December 2006, 1015-1021.
[Google Scholar] [CrossRef] 
[34] Achlioptas, P., Diamanti, O., Mitliagkas, I. and Guibas, L. (2018) Learning Representations and Generative Models for 3d Point Clouds. 35th International Conference on Machine Learning (ICML), Stockholm, 10-15 July 2018, 40-49.
[35] Wu, Z., Song, S., Khosla, A., et al. (2015) 3D ShapeNets: A Deep Representation for Volumetric Shapes. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, 7-12 June 2015, 1912-1920.
[36] Chang, A.X., Funkhouser, T., Guibas, L.J., et al. (2015) ShapeNet: An Information-Rich 3D Model Repository.
[37] Lian, Z., Zhang, J., Choi, S., et al. (2015) Non-Rigid 3D Shape Retrieval. Proceedings of the 2015 Eurographics Workshop on 3D Object Retrieval, Zurich, 2-3 May 2015, 107-120.
[38] Sketchfab.
https://sketchfab.com
[39] Yang, F.Z., Yang, H., Fu, J.L., et al. (2020) Learning Texture Transformer Network for Image Super-Resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, 13-19 June 2020, 5791- 5800.
[Google Scholar] [CrossRef] 
[40] Chen, H.T., Wang, Y.H., Guo, T.Y., et al. (2021) Pre-Trained Image Processing Transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, 20-25 June 2021, 12299-12310.
[Google Scholar] [CrossRef] 
[41] Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2020) An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.
[42] Liu, Z., Lin, Y.T., Cao, Y., et al. (2021) Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, 10-17 October 2021, 10012-10022.
[Google Scholar] [CrossRef] 
[43] Carion, N., Massa, F., Synnaeve, G., et al. (2020) End-to-End Object Detection with Transformers. In: European Conference on Computer Vision, Springer, Berlin, 213-229.
[Google Scholar] [CrossRef] 
[44] Zhu, X.Z., Su, W.J., Lu, L.W., et al. (2020) Deformable DETR: Deformable Transformers for End-to-End Object Detection.
[45] Qiu, S., Anwar, S. and Barnes, N. (2021) Pu-Transformer: Point Cloud Upsampling Transformer. Proceedings of the Asian Conference on Computer Vision, Macau, 4-8 December 2022, 2475-2493.