摘要
针对校园各物联网应用系统处理海量数据的性能差、数据的存储和运维成本高以及设备扩容升级困难等问题,设计了一种基于Hadoop的数据处理系统,为构建校园云数据中心、实现校园的智慧化服务提供有益的参考方案.文件处理模块针对海量结构化小文件的处理需求提出改进方案,对比实验表明该方案在降低集群主节点的内存消耗和提高小文件访问效率方面优于现有方案.
According to the problems of poor performance of massive data processing, high cost for storage, operation and maintenance, difficult expansion and upgrading of equipment existing in the single campus application system, a data proc- essing system was designed based on Hadoop, which can provide a useful scheme for building campus cloud data center and realizing campus wisdom service. A feasible improvement program of the system file processing module was proposed in order to meet the processing needs of massive structured small files. Experiments show that the improved program is more efficient than the existing solutions to reducing the memory consumption of the primary node and can improve the accessing efficiency of small files.
出处
《天津科技大学学报》
CAS
北大核心
2015年第5期72-77,共6页
Journal of Tianjin University of Science & Technology
基金
天津市科技型中小企业创新资金资助项目(12ZXCXGX33500)
国家自然科学基金资助项目(61272509)
