一种基于知识蒸馏与交叉注意力的轻量级多模态增强子识别方法

A Lightweight Multimodal Enhancer Identification Method Based on Knowledge Distillation and Cross-Attention

摘要: 增强子是一类非编码元件，在基因转录调控中起关键作用。如果增强子出问题，会和很多疾病密切相关。传统实验方法在鉴定增强子时，成本高，时间也长。现有的计算方法在处理单细胞数据或者覆盖度较低的表观组学数据时，识别效果会明显变差。我们提出了一种用于低样本量表观基因组学增强子识别的知识蒸馏轻量化多模态交叉注意力模型(AttLight-Enhancer)，这是一个多模态轻量化深度学习模型，用到了交叉注意力机制。它的目标是从低覆盖度多组学数据中识别增强子。我们还用了知识蒸馏的方法，把复杂教师模型里的知识迁移到轻量化学生网络中。我们在模拟低覆盖度数据和真实单细胞ATAC-seq数据上做了实验。模拟数据的覆盖度从10%一直降到1%。结果显示，当覆盖度只有1%时，AttLight-Enhancer的AUROC达到了0.865，比各个基线模型都好。经过知识蒸馏，学生模型的参数量只有教师模型的18.5%。通过注意力可视化分析，我们还看到了不同表观遗传特征的重要程度。其中H3K27ac最重要，然后是染色质可及性，最后是序列信息。

Abstract: Enhancers are a class of non-coding elements that play a critical role in the transcriptional regulation of genes. Enhancer dysfunction is closely associated with numerous diseases. Traditional experimental methods for identifying enhancers are costly and time-consuming. Existing computational approaches exhibit significantly degraded performance when processing single-cell data or epigenomic data with low coverage. In this study, we propose AttLight-Enhancer, a lightweight multimodal deep learning model that employs a cross-attention mechanism. It is designed to identify enhancers from low-coverage multi-omics data. We also utilize knowledge distillation to transfer knowledge from a complex teacher model to a lightweight student network. We conduct experiments on both simulated low-coverage data and real single-cell ATAC-seq data, with the coverage of simulated data reduced from 10% to 1%. The results demonstrate that AttLight-Enhancer achieves an AUROC of 0.865 at only 1% coverage, outperforming all baseline models. After knowledge distillation, the number of parameters of the student model is merely 18.5% of that of the teacher model. Through attention visualization analysis, we further reveal the importance of different epigenetic features: H3K27ac is the most critical, followed by chromatin accessibility, and finally sequence information.