一种阶段重置的知识蒸馏方法研究与仿真

Design and Simulation of Stage Reset Knowledge Distillation Method

摘要: 知识蒸馏是一种将知识从教师网络传递到学生网络的模型压缩方法。目前的知识蒸馏方法存在教师网络和学生网络之间的语义信息不一致的问题，具体而言，师生模型之间的前向推理距离不一致导致语义信息不一致，最终损耗蒸馏性能。为了解决这个问题，本文探索一种新的阶段重置知识蒸馏方法。该方法设计了以阶段为单位的知识蒸馏，师生网络相同阶段共享输出，降低了由学生与教师推理路径长度差异过大造成的特征语义不匹配的影响，从而提升学生网络的性能。最后，本文用提出的方法在公共数据集上进行仿真实验，并与最新的方法进行比较，实验结果表明本文提出的方法更具优势。

Abstract: Knowledge distillation is a compression technique used to transfer knowledge from a teacher net-work to a student network. However, the current knowledge distillation methods suffer from an is-sue of inconsistent semantic information between the teacher and student networks. This incon-sistency arises due to variations in forward reasoning distance between the teacher-student model, resulting in a loss of distillation performance. To address this problem, this study introduces a novel approach called “stage reset knowledge distillation.” This method incorporates stage-based knowledge distillation, where the output is shared within the same stage of the teacher-student network, which reduced the influence of feature semantic mismatch caused by the large difference in reasoning path length between students and teachers, thus enhancing the performance of the student network. Experimental evaluations on a public dataset are conducted to validate the pro-posed method’s efficacy. Comparative analysis against state-of-the-art techniques demonstrates the superior advantages offered by the proposed method.