一种自适应的分布式深度神经网络推理框架

An Adaptive Distributed Deep Neural Network Inference Framework

摘要: 近年来，随着深度学习的发展，深度神经网络(Deep Neural Network, DNN)模型变得越来越复杂，所需的内存和数据传输量也随之增大，这不仅降低了DNN的训练和推理速度，也限制了DNN在一些内存较小、计算能力较差的物联网(Internet of Things, IoT)设备上的部署。现有研究将基于云–边–端协同的分布式计算框架与深度神经网络相结合，组成了分布式深度神经网络(Distributed Deep Neural Network, DDNN)框架，该框架在IoT应用场景下有着显著的优势。然而，DDNN框架存在设备的计算能力有限、以及设备之间的传输成本较高等问题。针对上述问题，本文提出了自适应的分布式深度神经网络(Adaptive Distributed Deep Neural Network, ADA-DDNN)推理框架。ADA-DDNN框架采用了多个边缘出口，这些边缘出口允许ADA-DDNN框架中的模型在不同的深度层次上进行自适应地推理，以适应不同的任务需求和数据特性。此外，该框架增加了额外的边缘处理模块，边缘处理模块可以在边缘端进行特征融合之前，判断每个终端模块的输出结果是否可信，若可信，则直接输出分类结果，无需进行特征融合和后续计算。这大大增加了样本的边缘出口概率，减少了后续的计算成本。本文在开放的CIFAR-10数据集上进行验证，实验结果表明，ADD-DDNN框架在保证云端测试精度的前提下，显著提升了边缘测试精度。

Abstract: In recent years, the development of deep learning has led to an increase in the complexity of deep neural network (DNN) models, accompanied by a proportional increase in the amount of memory and data transfer required. This has a detrimental effect on the training and inference speed of DNNs, as well as limiting their deployment on some Internet of Things (IoT) devices with limited memory and computational capabilities. Existing research combines a distributed computing framework based on cloud-edge-end collaboration with deep neural networks to form the Distributed Deep Neural Network (DDNN) framework, which has significant advantages in IoT application scenarios. However, the DDNN framework suffers from the problems of limited computational power of devices, as well as high transmission cost between devices. To address these issues, this paper proposes the Adaptive Distributed Deep Neural Network (ADA-DDNN) inference framework. The ADA-DDNN framework employs multiple edge exits, which allow the models in the ADA-DDNN framework to perform different depth levels of adaptive reasoning to accommodate different task requirements and data characteristics. Furthermore, the framework incorporates an additional edge processing module, which is responsible for evaluating the credibility of the output results produced by each terminal module. If the results are deemed credible, the classification results are directly outputted without feature fusion and subsequent computation. This significantly increases the probability of the sample being outputted at the edge and reduces the subsequent computation cost. The paper validates the ADD-DDNN framework on the open CIFAR-10 dataset. The experimental results demonstrate that the framework significantly improves edge testing accuracy while maintaining the same level of testing accuracy in the cloud.