基于Spark计算框架的分布式文本分类方法研究

A Comparative Study of Distributed Text Categorization Methods Based on Different Computing Frameworks

摘要: 针对传统文本分类算法在面对日益增多的海量文本数据时效率低下的问题，论文在Spark计算框架上设计并实现了一种并行化朴素贝叶斯文本分类器，并着重介绍了基于Spark计算框架的文本分类实现过程。实验阶段，分别在单机、Map Reduce和Spark三种不同的计算框架下测试了文本分类的效率，并使用控制变量的方法在Spark计算框架下设计对照实验。实验证明，Spark计算框架下的朴素贝叶斯算法在面对海量文本分类时有着较高的处理效率。

Abstract: In view of the low efficiency of traditional text classification algorithms in the face of increasing mass of text data, a parallel naive Bayes text classifier is designed and implemented on the Spark computing framework, and the implementation process of text classification based on Spark computing framework is introduced. In the experimental stage, the efficiency of text classification is tested under three different computing frameworks, single machine, Map Reduce and Spark, and control experiments are designed under the Spark computing framework using control variables. Experiments show that naive Bayes algorithm in Spark computing framework has high efficiency in dealing with massive text categorization.