基于Spark的流程对象并行数据挖掘的研究与实现
Research and Implementation of Parallel Data Mining of Process Object Based on Spark
DOI: 10.12677/HJDM.2016.64018, PDF,   
作者: 郑雅飞, 杜 韬, 曲守宁：济南大学信息科学与工程学院，山东 济南；朱连江：济南大学信息网络中心，山东 济南
关键词: 数据挖掘；并行计算；流程对象；Spark；MapReduce；Data Mining； Parallel Computing； Process Object； Spark； MapReduce
摘要: 本文研究了基于Spark的并行数据挖掘，并将其应用到了流程对象数据分析中。文章通过对串行的流程对象数据挖掘算法流的研究，提出了一种基于Spark并行计算框架的并行化算法流解决方案，并通过编程实现、并行效率测试、算法调优，最终得出一个并行效果良好的并行数据挖掘方案。该并行方案明显提高了计算效率。
Abstract: In this paper, we study the parallel data mining based on Spark, and apply it to the data analysis of process object. We propose some parallel algorithm flow solutions based on Spark by studying the algorithm flow of stand-alone process object data mining. Through programming, parallel efficiency testing and algorithm tuning, we conclude an optimized parallel algorithm flow. These solutions improve the computational efficiency.
文章引用：郑雅飞, 杜韬, 朱连江, 曲守宁. 基于Spark的流程对象并行数据挖掘的研究与实现[J]. 数据挖掘, 2016, 6(4): 158-167. https://dx.doi.org/10.12677/HJDM.2016.64018

参考文献

[1] Gropp, W., Lusk, E., Doss, N., et al. (1996) A High-Performance, Portable Implementation of the MPI Message Passing Interface Standard. Parallel Computing, 22, 789-828. http://dx.doi.org/10.1016/0167-8191(96)00024-5 [Google Scholar] [CrossRef] 
[2] Reyes-Ortiz, J.L., Oneto, L. and Anguita, D. (2015) Big Data Analytics in the Cloud: Spark on Hadoop vs MPI/ OpenMP on Beowulf ☆. Procedia Computer Science, 53, 121-130. http://dx.doi.org/10.1016/j.procs.2015.07.286 [Google Scholar] [CrossRef] 
[3] Zaharia, M., Chowdhury, M., Franklin, M.J., et al. (2010) Spark: Cluster Computing with Working Sets. Proceedings of the 2nd USENIX conference on Hot Topics in Cloud Computing, USENIX Association, 10.
[4] Apache Spark. http://spark.apache.org
[5] Dean, J. and Ghemawat, S. (2004) MapReduce: Simplified Data Processing on Large Clusters. Proceedings of Operating Systems Design and Implementation (OSDI), 51, 107-113.
[6] Armbrust, M., Das, T., Davidson, A., et al. (2015) Scaling Spark in the Real World: Performance and Usability. Pro-ceedings of the VLDB Endowment, 8, 1840-1843. http://dx.doi.org/10.14778/2824032.2824080 [Google Scholar] [CrossRef] 
[7] Zaharia, M., Chowdhury, M., Das, T., et al. (2012) Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation, USENIX Association, 2.
[8] Song, Q., Guo, Q., Wang, K., et al. (2014) A Scheme for Mining State Association Rules of Process Object Based on Big Data. Journal of Computer and Communications, 2, 17-24. http://dx.doi.org/10.4236/jcc.2014.214002 [Google Scholar] [CrossRef] 
[9] Yu, H., Wen, J., Wang, H., et al. (2011) An Improved Apriori Algorithm Based On the Boolean Matrix and Hadoop. Procedia Engineering, 15, 1827-1831. http://dx.doi.org/10.1016/j.proeng.2011.08.340 [Google Scholar] [CrossRef] 
[10] 朱连江, 马炳先, 赵学泉. 基于轮廓系数的聚类有效性分析[J]. 计算机应用, 2010(S2): 139-141+198.

为你推荐