基于随机森林算法的盖亚大数据清洗的研究

An Automatic Data Cleaning Method for GPS Trajectory Data on Didi Chuxing GAIA Open Dataset Using Random Forest Algorithm

本文针对滴滴出行的盖亚开放数据集中的GPS轨迹数据,设计了一种自动数据清洗方法。该方法基于随机森林算法,用来识别盖亚开放数据集原始GPS轨迹数据中的无效数据,弱信号数据和正常数据。首先根据三类数据的数学特征选择其特征集,然后确定其最优的特征子集维度。最后,基于python的pandas和scikit-learn实现所提出的方法,并以盖亚数据集中的2016年10~11月成都市二环数据集作为样本进行了实验来验证该数据清洗方法的效果,结果表明了该方法可以有效地完成弱信号数据和无效数据的数据清洗工作。

A new data cleaning method for the GPS trajectory data on Didi Chuxing GAIA Open Dataset is developed.Random forests algorithm is employed to the identification of invalid,weak,and normal data of the Didi Chuxing GAIA Open Dataset raw data.Firstly,the feature set is selected according to the mathematical characteristics of three types of data,and then the optimal feature subset dimension is determined.Finally,to implement the proposed method,the Pandas and scikit-learn Python library are used to read and process the data and the result illustrates the effectiveness of this method.