期刊文献+

面向电信应急系统的Spark性能预测与参数调优方法探究 被引量:2

Research on spark performance prediction and parameter optimization method for telecom emergency system
下载PDF
导出
摘要 在众多大数据处理平台中,Spark以其运算速度快,操作方便等特有的优势脱颖而出。然而在实际应用中,Spark还存在一定的优化空间。针对Spark的参数配置优化,基于随机森林回归算法提出一种可行性较高的Spark性能预测的方法,并将该方法应用到电信数据处理任务中。实验证实,该性能预测方法不仅易于实现,且通过该方法得到的参数配置对于Spark任务性能有较好的提升。
出处 《电信快报》 2020年第12期26-30,共5页 Telecommunications Information
  • 相关文献

参考文献3

二级参考文献21

  • 1White T. Hadoop: The definitive guide[J]. O'reilly Media Inc Gravenstein Highway North,2010,215(11):1-4.
  • 2Lakshman A,Malik P. Cassandra..A decentralized structured storage system[J]. Acre Sigops Operating Systems Review, 2010,44(2) :35-40.
  • 3Zaharia M,Chowdhury M,Franklin M J,et al. Spark:Cluster computing with working sets[C]//Proc of the 2nd USENIX Conference on Hot Topics in Cloud Computing, 2010:1765- 1773.
  • 4Seo S, Jang I, Woo K, et al. HPMR: Prefetching and pre- shuffling in shared MapReduce computation envlronment[C] //Proc of the 2009 IEEE International Conference on Cluster Computing, 2009 : 1-8.
  • 5Jiang D,Ooi B C, Shi L, et al. The performance of MapRe- duce:An in-depth study[J]. Proceedings of the VLDB En- dowment, 2010,3 (12) : 472-483.
  • 6Dittrich J. Hadoopq-q- :Making a yellow elephant run like a cheetah (without it even noticing)[J]. Proceedings of the VLDB Endowment, 2010,3 (12) : 518-529.
  • 7Shivnath B. Towards automatic optimization of MapReduce programs[C]//Proc of the 1st ACM Symposium on Cloud Computing, 2010 : 137-142.
  • 8Herodotou H,Lim H, Luo G, et al. Starfish: A self-tuning system for big data analytics[C]//Proc of the 5th Cidr Conf, 2011 : 261-272.
  • 9Shi Ju-wei,Zhou Jia, Lu Jia-heng, et al. MRTuner:A toolkit to enable holistic optimization for MapReduce )obs[C]//Proc of the VLDB Endowment, 2014,7(13) : 1319-1330.
  • 10Aaron D, Andrew O. Optimizing shuffle performance in spark [R]. CA: Berkeley-Department of Electrical Engineering and Computer Sciences, University of California, 2033.

共引文献22

同被引文献27

引证文献2

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部