基于Hadoop的云存储系统文件处理与安全研究

Research on file processing and security of cloud storage system based on Hadoop

由于公有云存储中存在数据泄露、丢失、存储不稳定等不安全因素,私有云存储成为解决当前企业安全存储需求的最佳选择。针对单点内存负载过高,写入效率低下的问题,设计了一套处理大量小文件的模块,通过缓存多个小文件合并,再合并文件元数据放置索引表,通过索引表中的偏移量寻找块中小文件数据的方式,提高HDFS写入大量小文件的系统性能,经过测试验证了优化方案的有效性。

Since the public cloud storage exisits the unsafe factors of data leakage, data lose and unstable storage, the pri- vate cloud storage becomes the best choice to meet the current safety storage demands of the enterprises. To solve the too high load of the single point memory and low writing efficiency, a module to deal with the massive small files was designed. Multiple small files are merged through the cache, and the index table is placed into the metadata of the emerged file. The method of finding small file data in the block by means of the offset in the index table can improve the HDFS performance of writing mas- sive small files. The validity of the optimization scheme was verified with an experiment.