基于Hadoop平台的交通管理数据存储系统设计分析

Design and Analysis of Traffic Management Data Storage System Based on Hadoop Platform

信息大爆炸产生的数据数量非常庞大且复杂,其中不仅有结构化与半结构化的数据,还有着非结构化的文本、图像、视频等信息数据。因此,对于大数据的分析也成为了研究的重点,尤其是大数据的采集、处理、存储、分析以及安全,是研究的主要目标。在大数据的处理中,Hadoop作为系统基础平台能够实现大数据的分布式处理,其具备可靠、安全、高效以及可伸缩的特点。HDFS是Hadoop平台的基本组件,同时也是目前研究与应用非常广的一种存储大数据的分布式文件系统,而其根本是分流式的分块文件系统。而传统的HDFS读取较大流量文件数据时会出现I/O的性能问题,基于针对此问题进行分析研究进而提出优化后的HDFS存取策略。借助于文件的预处理,实现文件的汇总,提出新的元数据管理与通用文件存储模式,解决HDFS处理大量文件时出现的性能问题,满足交通应用的大数据处理和存储需求。

The amount of data generated by the information explosion was very large,including not only structured and semi-structured data,but also unstructured text,image,video and other information data.Therefore,the analysis of big data had become into the focus of research,especially the collection,processing,storage,analysis and security of big data,which was the main goal of research.during the data processing,Hadoop,as the basic platform of the system,could realize the distributed processing of big data,which had the characteristics of reliability,security,efficiency and scalability.HDFS was a basic component of Hadoop platform.It was also a distributed file system for storing big data that was widely studied and applied at present.It was basically a split block file system.However,I/O problems occured when traditional HDFS reads big data.Based on the analysis and research of this problem,an optimized HDFS access strategy was proposed.With the help of file preprocessing,the summary of files was realized,and a new metadata management and general file storage mode was proposed to solve the I/O problems of HDFS with the big data,and meet the requirements of traffic applications for big data processing and storage.