基于分布式计算框架的机器学习系统分析

Analysis of Machine Learning System Based on Distributed Computing Framework

针对机器学习对大数据处理的消耗大量存储空间和时间的问题,提出利用对数据分区存储和机器学习对分区数据并行处理的方法,对分布式计算框架的机器学习系统框架和矩阵运算对数据处理进行研究,并分析机器学习的模型和数据集分布式处理的方法,实验仿真结果表明,通过对大数据集进行分布式存储和并行计算,提高了机器学习的效率和对大数据的处理能力。

Aiming at the problem that machine learning consumes a lot of storage space and time for big data processing,this paper proposes the method of partitioned data storage and parallel processing of partitioned data by machine learning,and studies the machine learning system framework and matrix operation of distributed computing framework.By analyzing the model of machine learning,the method of distributed processing of data sets and the experimental simulation results,it shows that the efficiency of machine learning and the processing ability of big data are improved by distributed storage and parallel computing of large data sets.