基于Hadoop的海量图片存储平台的设计与开发

Design and Development of the Mass Image Storage Platform Based on Hadoop

随着Internet的飞速发展与深入应用,海量图片数据的存取问题显得越发突出,传统存储架构已突显管理效率不高、存储能力不足及成本太高等问题,Hadoop为我们提供了一种新的解决问题的思路,Hadoop可以充分利用集群的威力进行高速运算和存储,但是小文件过多时Hadoop的Name Node将导致内存出现瓶颈问题,使得系统效率变得极为低下。该文提出了一种基于Hadoop的、可对海量图片文件进行高效处理的存储架构,通过预处理模块的归类算法,并引入扩展一级索引机制,能较好地解决海量图片的处理问题,并避免内存瓶颈问题。实验表明,该系统易维护、具有良好的可扩展性,其稳定性、安全性、并发性均有较大改善。

With fast development and deep appliance of the Internet,problem of mass image data storage stand out,so the problem of low management efficiency,low storage ability and high cost of traditional storage framework has appeared.The appearance of Hadoop provides a new thought.However,Hadoop itself is not suit for the handle of small files.This paper puts forward a storage framework of mass image files based on Hadoop,and solved the internal storage bottleneck of Name Node when small files are excessive through classification algorithm of preprocessing module and lead-in of high efficiency and first-level of index mechanism.The test manifests that the system is safe,easy to defend and has fine extension quality;as a result,it can reach to a fine effect.