基于Hadoop的ADS-B数据解析与存储方法

The Data Analysis and Storage of ADS-B Based on Hadoop

广播式自动相关监视系统(Automatic Dependent Surveillance-Broadcast,ADS-B)是国际民航组织(ICAO)推荐使用的集数据通信、卫星导航和监视技术于一体的新一代航空器运行监视系统,可以自动的接收和发送飞机及其周围的信息。随着监视区域内航班数量的增加,对于以秒为单位进行收发信息的ADSB而言,单机环境已经无法满足海量ADS-B数据的解析、存储与分析,本文利用Mapreduce模型提供的高效分布式编程和运行框架对ADS-B数据进行解析,将解析后的数据存储到基于Hive的ADS-B数据仓库,并通过Mysql建立的索引表联合Hive中的分桶操作对信息种类进行划分,有效提高了数据解析效率并避免了Hive中索引不完善引起的查询效率低的问题。实验表明对于海量的ADS-B数据,利用Mapreduce进行解析并利用Hive进行存储分析的效率明显提升。

Automatic dependent surveillance-broadcas（ ADS-B） is a new generation of aircraft operation monitoring system that integrates data communications,satellite navigation with surveillance technology recommended by the interbational civil aviation organization（ ICAO） can receive and send the information of the aircraft and it＇s surrounding automatically. With the increasing number of flights within the surveillance area,for the ADS-B in seconds to send and receive information,the stand-alone environment has been unable to meet the massive ADS-B data analysis,storage and analysis. In this paper,the ADS-B data is parsed by using the efficient distributed programming and runtime framework provided by the Mapreduce model. The parsed data is stored in the ADS-B data warehouse based on index table created by Mysql combined with the bucket in Hive operation to classify the message,which effectively improves the efficiency of data analysis and avoids the problem that the query efficiency caused by imperfect index in Hive is low.Regarding the massive and historical data of ADS-B,it is proven that the parsing by map-reduce frame and efficiency of storage analysis by Hive is promoted obviously.