基于MongoDb的分布式高并发日志系统的设计

Design of Distributed and High Concurrency Log System Based on MongoDb

摘要: 日志系统是一个完整信息系统的重要组成部分，海量的日志信息一方面可以挖掘出用户的通用行为模式，提高系统的服务质量，另一方面还为系统优化，系统结构调整提供了依据，在面对高并发量的系统访问时，及时准确的收集日志成了日志分析的关键，传统日志信息的记录与业务系统代码交织，耦合过高，日志的记录因占用系统的资源进而影响正常业务系统的运行。为了改善这一现状，本文提出了一种与原业务系统零耦合的新设计方式，该方式从网络请求的负载均衡、分布式集群服务、服务中的多线程处理、和非关系型数据库MongoDb的分片拓展四个维度展开，实现了一套异步、分布、并行处理的日志记录系统，后经千万级数据量的压测，从系统吞吐能力，系统内存占用情况等多个方面证实了设计的可行性与强稳定性。

Abstract: Logging system is an important part of a complete information system. On one hand, massive log information can dig out the common behavior patterns of users to improve service quality; on the other hand, it provides a theoretical basis for system optimization and system structural adjust-ment. When faced with a high concurrency value system access, timely and accurate collection logs are the key for log analysis. Traditional log records of information have a high coupling with busi-ness system code. It affects normal business systems because of taking up system resources. In order to improve this situation, this paper presents a new design way which has zero coupling with an original business system. The way realizes a asynchronous, distributed, parallel pro- cessing log records by four parts, which are network request load balancing, distributed cluster services, services in the multi thread processing, and non relational database MongoDb shard expand. We confirmed the feasibility of the design and strong stability by analysised system memory usage and system throughput capacity after tens of millions of data compression test.