一种基于动态水位值的Flink调度优化算法
A Flink Scheduling Optimization Algorithm Based on Dynamic Water Level
DOI: 10.12677/CSA.2021.115155, PDF,    科研立项经费支持
作者: 冯 鹏：大连大学信息工程学院，辽宁 大连;大数据应用技术国家民委重点实验室，辽宁 大连；黄 山, 段晓东：大数据应用技术国家民委重点实验室，辽宁 大连;大连民族大学计算机科学与工程学院，辽宁 大连;大连市民族文化数字技术重点实验室，辽宁 大连
关键词: Flink；大数据；反压；Netty；Flink； Big Data； Backpressure； Netty
摘要: 新一代大数据间引擎Flink在面临远程传输问题时，主要通过Netty完成数据传输，并依靠Netty水位值机制来保证其反压机制的运行。Netty水位值机制是一种相对静态的机制，这使得Flink在面临突变性特别大的数据流时会反复进行反压，进而影响整个Flink集群的计算效率。针对此问题，本文提出一种基于动态水位值的Flink调度优化算法Flink-N，经实验验证，与Flink默认的反压机制相比，Flink-N在吐吞量、CPU利用率及时延均有很大提升，时延整体优化达18%，最高优化23%。
Abstract: When the new generation of big data engine Flink is faced with the problem of remote transmission, it mainly completes the data transmission through Netty, and relies on the Netty water level mechanism to ensure the operation of its back pressure mechanism. Netty water level mechanism is a relatively static mechanism, which makes Flink repeatedly back pressure in the face of catastrophic data flow, thus affecting the computing efficiency of the whole Flink cluster. To solve this problem, this paper proposes a Flink scheduling optimization algorithm Flink-N based on dynamic water level. Experimental results show that, compared with Flink’s default back pressure mechanism, Flink-N greatly improves the throughput, CPU utilization and time delay. The overall delay optimization is 18%, and the maximum optimization is 23%.
文章引用：冯鹏, 黄山, 段晓东. 一种基于动态水位值的Flink调度优化算法[J]. 计算机科学与应用, 2021, 11(5): 1512-1521. https://doi.org/10.12677/CSA.2021.115155

参考文献

[1] 袁海飞. 基于分布式实时计算架构的生产设备数据分析平台[J]. 电子技术与软件工程, 2020, 174(4): 217-219.
[2] 樊春美, 朱建生, 单杏花, 等. 基于Flink实时计算的自动化流控制算法[J]. 计算机技术与发展, 2020, 30(8): 66-72.
[3]
https://tech.meituan.com/2018/10/18/meishi-data-flink.html
[4]
https://help.aliyun.com/document_detail/110778.html?spm=a2c4g.11174283.2.6.114773d5r0epNT
[5] 姜红玉, 汪朋, 封雷. 基于流式计算的实时用户画像系统研究[J]. 计算机技术与发展, 2020, 30(7): 186-193.
[6] Aniello, L., Baldoni, R. and Querzoni, L. (2013) Adaptive Online Scheduling in Storm. Proceedings of the 7th ACM International Conference on Distributed Event-Based Systems, June 2013, 207-218. [Google Scholar] [CrossRef] 
[7] Yang, M. and Ma, R.T.B. (2015) Smooth Task Migration in Apache Storm. Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, May 2015, 2067-2068. [Google Scholar] [CrossRef] 
[8] 林琳. 阿里巴巴的大数据之路JStorm与Blink的发展史[J]. 计算机与网络, 2019, 45(2): 41-42.
[9]
https://developer.aliyun.com/article/709397?spm=a2c6h.14164896.0.0.b095505ax2TAav
[10] 宋灵城. Flink和Spark Streaming流式计算模型比较分析[J]. 通信技术, 2020, 53(1): 59-62.
[11] Mhand, M.A., Boulmakoul, A. and Badir, H. (2019) Scalable and Distributed Architecture Based on Apache Spark Streaming and PROM6 for Pro-cessing RoRo Terminals Logs. Proceedings of the New Challenges in Data Sciences: Acts of the Second Conference of the Moroccan Classification Society, March 2019, Article No. 19. [Google Scholar] [CrossRef] 
[12] 关沫, 魏碧晴. 基于Flink框架的TopN堆排序优化算法[J]. 信息技术与网络安全, 2020(2): 23-26.
[13] 汪丽娟, 钱育蓉, 张猛, 等. 基于Flink平台的资源感知任务调度策略[J]. 东北师大学报:自然科学版, 2020, 52(2): 66-72.
[14] 何贞贞, 于炯, 李梓杨, 等. 基于Flink的任务调度策略[J]. 计算机工程与设计, 2020, 41(5): 1280-1287.
[15] Chen, C., Li, K., Ouyang, A., et al. (2016) GFlink: An In-Memory Computing Architecture on Heterogeneous CPU-GPU Clusters for Big Data. 45th International Conference on Parallel Processing (ICPP), Philadelphia, 16-19 August 2016, 542-551. [Google Scholar] [CrossRef] 
[16] Chen, C., Li, K., Ouyang, A. and Li, K. (2018) FlinkCL: An OpenCL-Based In-Memory Computing Architecture on Hetero-geneous CPU-GPU Clusters for Big Data. IEEE Transactions on Computers, 67, 1765-1779. [Google Scholar] [CrossRef] 
[17] Chen, C., Li, K., Ouyang, A., et al. (2017) GPU-Accelerated Paral-lel Hierarchical Extreme Learning Machine on Flink for Big Data. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 47, 2740-2753. [Google Scholar] [CrossRef]