基于多模态融合驱动的智慧课堂感知系统实验方法设计

Design of Experimental Method for Smart Classroom Perception System Driven by Multimodal Fusion

摘要: 针对人工智能师范专业实验教学中单一模态训练局限、技术与教育场景脱节，以及学生课堂整体感知能力培养不足的问题，本文设计并实现了基于语音–视觉双模态融合的智慧课堂感知系统及对应实验。实验以“技术融合–场景适配–能力验证”为核心，整合Whisper、YOLO26、改进型ViT及大语言模型，构建全流程实验体系，实现教学阶段划分、学生个体识别与追踪、动作识别及多模态数据融合，输出课堂感知综合报告。结合前文实验流程与验证，结果表明，该方案规范可行，突破传统实验局限，可帮助师范生巩固核心技术、提升工程实践能力，培养课堂感知与场景适配能力，为智能教育复合型师资培养提供实践载体，也为相关专业实验教学体系优化提供参考。

Abstract: Aiming to address the problems of single-modal training limitations, the disconnect between technology and educational scenarios, and the insufficient cultivation of students’ overall classroom perception ability in the experimental teaching of AI-oriented teacher education programs, this paper designs and implements a smart classroom perception system based on audio-visual bimodal fusion, along with corresponding experiments. Centered on the framework of “technology integration-scenario adaptation-capability validation”, the experiments integrate Whisper, YOLO26, an improved Vision Transformer (ViT), and a large language model to build a full-process experimental system, achieving teaching phase classification, individual student identification and tracking, action recognition, and multimodal data fusion, and outputting a comprehensive classroom perception report. In conjunction with the experimental procedures and validations described earlier, the results show that the proposed scheme is standardized and feasible, breaking through the limitations of traditional experiments. It helps pre-service teachers consolidate core technologies, enhance engineering practice skills, and cultivate classroom perception and scenario adaptation abilities. This provides a practical platform for cultivating compound talents in intelligent education and offers a reference for optimizing experimental teaching systems in related disciplines.