云雾网络架构的大数据分析平台研究

Research on big data analysis platform of cloud network architecture

随着云计算和物联网的快速发展,企业和个人产生的数据信息量和速度都在不断增加。如何使用和管理如此庞大的数据量进行大数据分析已经成为一项重要的任务。但在云计算的架构下,Hadoop在执行MapReduce操作时会占用大量的网络带宽,尤其是在多个节点的情况下,网络传输质量的影响更为严重。基于大数据信息储存平台为基础架构,结合雾计算的数据预处理能力,构建了以雾计算架构为基础的大数据分析平台,利用雾的特性计算近端设备的优点,将大多数的数据信息在送进MapReduce做分析前,先进行初步的整合和计算,可以减少传递给Hadoop MapReduce的Result Set,将这种计算架构应用到当前的云计算环境中,达到精简网络资源的目的,进而提高大数据运算整体执行效率。

With the rapid development of cloud computing and Internet of Things,the amount and speed of data information generated by enterprises and individuals are increasing.How to use and manage such a huge amount of data for large data analysis has become an important task.However,in the framework of cloud computing,Hadoop will occupy a large amount of network bandwidth in MapReduce operation,especially in the case of multiple nodes,the impact of network transmission quality is more serious.Based on the large data information storage platform and the data preprocessing ability of fog computing,this paper constructs a large data analysis platform based on fog computing architecture.By utilizing the advantages of fog computing near end equipment,most of the data information is integrated and calculated preliminarily before being sent to MapReduce for analysis.Computing can reduce the Result Set passed to Hadoop MapReduce,and apply this computing architecture to the current cloud computing environment to streamline network resources,thereby improving the overall execution efficiency of large data operations.