期刊文献+

基于大数据技术的EAST实验数据访问日志分析系统的设计 被引量:2

DESIGN OF THE EAST EXPERIMENTAL DATA ACCESS LOG ANALYSIS SYSTEM BASED ON BIG DATA TECHNOLOGY
下载PDF
导出
摘要 EAST装置产生的实验数据规模日益变大,对EAST上的MDSplus数据存储服务器进行有效地监控是很有必要的。为了方便实验人员对MDSplus服务器上的用户进行管理,设计一个MDSplus日志离线和实时分析系统。MDSplus日志分析系统采用的大数据处理框架是Hadoop生态圈的MapReduce离线计算模型和Spark生态圈中的Spark Streaming实时数据计算模型。系统还使用Flume、Kafka的日志监测、聚合、分发等关键性技术,使得MDSplus海量日志数据的处理能力变为可能,且能够在秒级别处理千万条未处理的MDSplus日志信息,离线和实时处理后展现在Web端。测试表明,系统工作能够满足设计需求,对聚变实验数据的管理具有重要的应用价值。 The experiment data generated by the EAST device is getting larger and larger, and it is necessary to monitor the MDSplus data storage server on EAST. In order to facilitate the management of users on the MDSplus server, an MDSplus log offline and real-time analysis system is required. The big data processing frameworks, adopted by the MDSplus log analysis system, were the MapReduce offline computing model in the Hadoop ecosystem and the Spark Streaming real-time data computing model in the Spark ecosystem. The framework also made use of key technologies such as log monitoring, aggregation and distribution with framework likes Flume and Kafka, which made it possible for MDSplus mass log data processing power. The system could process tens of millions of unprocessed MDSplus log information at a second level, and then display it on the web after offline and real-time processing. The test shows that the system can meet the design requirements and has important application value to the management of fusion experiment data.
作者 章琦皓 王枫 王月婷 Zhang Qihao;Wang Feng;Wang Yueting(Institute of Plasma Physics,Chinese Academy of Sciences,Hefei 230031,Anhui,China;University of Science and Technology of China,Hefei 230026,Anhui,China)
出处 《计算机应用与软件》 北大核心 2018年第9期50-55,共6页 Computer Applications and Software
基金 国家重点研发计划项目(2017YFE0300500 2017YFE0300505)
关键词 MDSplus日志 HadoopMR模型 SPARK STREAMING FLUME Kafka MDSplus log HadoopMR model Spark Streaming Flume Kafka
  • 相关文献

参考文献6

二级参考文献17

  • 1周丽娟,王慧,王文伯,张宁.面向海量数据的并行KMeans算法[J].华中科技大学学报(自然科学版),2012,40(S1):150-152. 被引量:32
  • 2周海汉.HBase[EB/OL].http://hbaseapache.org,2013.04-07.
  • 3Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung. TheGoogle File System[F.B/OL].http://labs.google.com/pctpers/gfs-sosp2003. pdf, 2011-03-29.
  • 4怀特著.周敏奇,钱卫宁,金澈清,王晓玲译.Hadoop权威指南(第2版)[M].北京:清华大学出版社,2011.
  • 5guiii.hadoop作业调优参数整理及原理[EB/OL].http://www.tbdata.0rq/0rchjves/1470,2011-01-20.
  • 6Dhruba BOrthakur & Joydeep Sen Sarma etc.Apache Hadoop Goes Reaitime clt Facebook[EB/OL].http://wenku.baidu.com/ view/5blf48ef0975f46527d3e18b.html , 2011-06-12.
  • 7拉姆著.韩冀中译.Hadoop实战[M].北京:人民邮电出版社,2011.
  • 8Sumit Shrestha.Bulk importing Data into HBase[FTB/OLt.http://www. deerwalk.com/bulk importing_data, 2011-07-26.
  • 9Engel S J, Gilmartin B J, Bongort K, et al. Prognostics, the real issues involved with predicting life remaining[C]/// Proceedings of the IEEE Aerospace. Big Sky: IEEE, 2000 : 457 - 469.
  • 10Mathur A. Data Mining of aviation data for advancing health management [ C]// Proceedings Component and Systems Diagnostics, Prognostics, and Health Management. Orlando: SPIE, 2002:61 -71.

共引文献163

同被引文献33

引证文献2

二级引证文献3

相关作者

内容加载中请稍等...

相关机构

内容加载中请稍等...

相关主题

内容加载中请稍等...

浏览历史

内容加载中请稍等...
;
使用帮助 返回顶部