基于FPGA的高效数据过滤技术研究

Research on Efficient Data Filtering Based on FPGA

为了提高Hadoop的数据处理效率,在Hadoop集群中加入FPGA对Hadoop分布式文件系统中存储的数据进行过滤,这样MapReduce程序只需要对过滤后的数据进行自定义的操作,从而避免了扫描全部数据.实验表明处理海量的数据时,FPGA能够显著地过滤掉无用数据,从而提高数据处理效率.

In order to increase the efficiency of data processing, FPGA was injected into Hadoop cluster. FPGA will filter the data stored in HDFS（Hadoop Distributed File System）. In this way MapRedeuce just need to process the filtered data. A series experiment shows that FPGA could filter useless data and increase the efficiency of data processing.