基于Spark的动态聚类算法研究

Research on Dynamic Clustering Algorithm Based on Spark Framework

摘要: 针对数据流的聚类算法，近年来取得了有效的进展，出现了许多卓有成效的算法。随着信息采集技术的进步，需要处理的数据量越来越大，需要研究针对数据流的并行聚类算法。本文基于串行的数据流聚类算法D-Stream作出并行化改进，用通用的大数据处理框架Spark设计了一个基于分布式架构运行的动态数据聚类算法PDStream。实验结果表明，该算法具有更高的效率和良好的扩展性，能够实现分布式架构下的流数据动态聚类。

Abstract: In the era of big data, with the rapid growth of data size, the requirements of data processing in-crease constantly. It has put forward many effective algorithms for data stream clustering these years. However, with the continuous development of social technology, single machine environ-ment has been difficult to meet the needs of data mining. Cluster environment is used more for information collection and data processing, the traditional clustering algorithm does not adapt well to the new processing requirements. This paper made some improvements from the data stream clustering algorithm D-Stream, used the big data processing framework Spark and designed a dynamic data clustering algorithm PDStream based on distributed architecture. The new algorithm is proved to be more efficient and able to perform dynamic clustering tasks under distributed architecture from the results of experiment.