智能终端海量数据采集与实时分析设计和应用研究

Design and Application of Massive Data Acquisition and Real-Time Analysis of Intelligent Terminal

摘要: 针对各种场景下的海量终端的部署，本文设计了一套海量数据采集与实时分析系统，具体在数据采集模块，通过借助Kafka消息队列，实现数据的高并发接入；在数据分析模块，借助大数据流处理系统Storm，在保证高可靠性的前提下，实现数据的实时处理，并通过相应的优化设计，解决海量终端接入网络时的高并发访问与数据处理需求；通过可视化设计以及实验验证本文方法的有效性，系统具有低延迟，高吞吐，可拓展等特点，能够满足车联网海量数据处理要求，具有很强的实用价值，目前本文提出的方法已经应用在实际场景中，为20多万台北斗定位终端提供服务。

Abstract: For the deployment of mass terminals in various scenarios, this paper designs a set of mass data acquisition and real-time analysis system. In the data acquisition module, with the help of Kafka message queue, the high concurrent access of data is realized; in the data analysis module, with the help of the big data stream processing system storm, the real-time data processing is realized on the premise of high reliability, and through the corresponding optimization design, the high concurrent access and data processing requirements of massive terminals accessing the network are solved; through the visual design and experimental verification, the effectiveness of this method, the system has the characteristics of low latency, high throughput, scalability, and can meet the requirements of massive data processing in the Internet of vehicles, which has strong practical value. At present, the method proposed in this paper has been applied in the actual scene, providing services for more than 200,000 BD-based positioning terminals.