一种基于spark的边缘云大数据分析算法研究

A Spark-based algorithm for analyzing big data in edge cloud

算力和时延等要求不断提高,加速了边缘计算(MEC)的规模应用,边缘计算将计算和存储功能下沉至网络边缘的数据生产侧,将传统云计算的部分处理任务迁移至边缘计算节点,提升了用户体验,但同时,分布式计算产生海量日志数据,对运维管理提出了更高要求。文章以云宽带边缘大数据为例,探索并提出了一种基于spark的日志数据的分析算法实现,并对比了Hadoop MapReduce在分析过程中的效率,该实现能够有效解决边缘计算中普遍存在的度量分析效率的问题。

The increasing demand for computing power and reduced latency has accelerated the scale of edge computing applications.Edge computing brings computation and storage capabilities to the data production side at the network edge,migrating certain processing tasks from traditional cloud computing to edge computing nodes.This improves user experience.However,distributed computing generates massive log data,placing higher demands on operations and management.This article takes cloud broadband edge big data as an example and explores and proposes an algorithm implementation for analyzing log data based on Spark.The efficiency of Hadoop MapReduce in the analysis process is also compared.The implementation effectively addresses the common problem of measurement analysis efficiency in edge computing.