基于多位置轻量适配器的文生图细节优化方法
Detail Enhancement Method for Text-to-Image Based on Multi-Position Lightweight Adapters
摘要: 目前，人工智能技术的快速发展带来了庞大的应用需求，图像生成已成为计算机视觉领域的重要研究方向。扩散模型凭借其优异的生成性能和稳定可控的训练过程，成为当前生成高质量图像的主流框架。然而，扩散模型仍面临细节保真度不足的问题，这限制了其在定制化应用中的灵活性与可控性。因此，如何在维持模型轻量化的同时有效提升生成质量与鲁棒性，成为扩散模型高效适配的核心挑战。针对通用文本到图像生成中存在的空间细节模糊与语义对齐偏差，本文设计了基于多位置轻量适配器的细节增强架构。该架构在残差块的输出端嵌入EBlock适配器，利用深度可分离卷积增强局部空间细节；在注意力层输出后接入DAT适配器，通过低秩映射对通道特征进行调制以提升文本–图像语义对齐。两类适配器均采用零初始化与渐进式激活机制，并结合知识蒸馏实现原始模型知识保留。该架构仅引入约0.35%的额外参数，在COCO2017数据集上Laplacian方差和FID分别为0.032和17.8；在Flickr30k数据集上FID进一步降至16.3，显著提升了生成图像的细节质量与语义一致性。
Abstract: Currently, the rapid development of artificial intelligence technology has brought about substantial application demands, and image generation has become an important research direction in the field of computer vision. Diffusion models, with their excellent generation performance and stable and controllable training process, have become the mainstream framework for generating high-quality images. However, diffusion models still face the problem of insufficient detail fidelity, which limits their flexibility and controllability in customized applications. Therefore, how to effectively improve generation quality and robustness while maintaining model lightweightness has become the core challenge for efficient adaptation of diffusion models. Addressing the spatial detail blurring and semantic alignment deviation in general text-to-image generation, this paper designs a detail enhancement architecture based on multi-position lightweight adapters. This architecture embeds an EBlock adapter at the output of the residual block to enhance local spatial details using depthwise separable convolution; it also incorporates a DAT adapter after the attention layer output to modulate channel features through low-rank mapping to improve text-image semantic alignment. Both adapters adopt zero initialization and progressive activation mechanisms, and combine knowledge distillation to preserve the knowledge of the original model. This architecture introduces only about 0.35% additional parameters, achieving a Laplacian variance and FID of 0.032 and 17.8 on the COCO2017 dataset, respectively; on the Flickr30k dataset, the FID further decreases to 16.3, significantly improving the detail quality and semantic consistency of generated images.
文章引用：马宇航. 基于多位置轻量适配器的文生图细节优化方法[J]. 人工智能与机器人研究, 2026, 15(3): 832-843. https://doi.org/10.12677/airr.2026.153077

参考文献

[1] Kingma, D.P. and Welling, M. (2013) Auto-Encoding Variational Bayes. [Google Scholar] [CrossRef] 
[2] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., et al. (2014) Generative Adversarial Networks. 2014 Advances in Neural Information Processing Systems, Montreal, 8-13 December 2014. [Google Scholar] [CrossRef] 
[3] Zhang, L., Rao, A. and Agrawala, M. (2023) Adding Conditional Control to Text-to-Image Diffusion Models. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Paris, 1-6 October 2023, 3813-3824. [Google Scholar] [CrossRef] 
[4] Ye, H., Zhang, J., Liu, S., et al. (2023) IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models. [Google Scholar] [CrossRef] 
[5] Wang, Q., Bai, X., Wang, H., et al. (2024) InstantID: Zero-Shot Identity-Preserving Generation in Seconds. [Google Scholar] [CrossRef] 
[6] Li, M., Yang, T., Kuang, H., et al. (2024) Controlnet++: Improving Conditional Controls with Efficient Consistency Feedback. In: Lecture Notes in Computer Science, Springer, 129-147. [Google Scholar] [CrossRef] 
[7] Feijoo, D., Benito, J.C., Garcia, A. and Conde, M.V. (2025) DarkIR: Robust Low-Light Image Restoration. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, 10-17 June 2025, 10879-10889. [Google Scholar] [CrossRef] 
[8] Chollet, F. (2017) Xception: Deep Learning with Depthwise Separable Convolutions. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 21-26 July 2017, 1800-1807. [Google Scholar] [CrossRef] 
[9] Lin, M., Chen, Q. and Yan, S. (2013) Network in Network. [Google Scholar] [CrossRef] 
[10] Han, K., Wang, Y., Guo, J. and Wu, E. (2024) ParameterNet: Parameters Are All You Need for Large-Scale Visual Pretraining of Mobile Networks. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, 16-22 June 2024, 15751-15761. [Google Scholar] [CrossRef] 
[11] Ba, J.L., Kiros, J.R. and Hinton, G.E. (2016) Layer Normalization. [Google Scholar] [CrossRef] 
[12] Aghajanyan, A., Gupta, S. and Zettlemoyer, L. (2021) Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, August 2021, 7319-7328. [Google Scholar] [CrossRef] 
[13] Hendrycks, D. and Gimpel, K. (2016) Gaussian Error Linear Units (GELUs). [Google Scholar] [CrossRef] 
[14] He, K., Zhang, X., Ren, S. and Sun, J. (2016) Identity Mappings in Deep Residual Networks. In: Lecture Notes in Computer Science, Springer, 630-645. [Google Scholar] [CrossRef] 
[15] Ho, J. and Salimans, T. (2022) Classifier-Free Diffusion Guidance. [Google Scholar] [CrossRef] 
[16] Pascanu, R., Mikolov, T. and Bengio, Y. (2013) On the Difficulty of Training Recurrent Neural Networks. 2013 International Conference on Machine Learning, Atlanta, 16-21 June 2013, 1310-1318.
[17] Hinton, G., Vinyals, O. and Dean, J. (2015) Distilling the Knowledge in a Neural Network. [Google Scholar] [CrossRef] 

为你推荐