云计算下海量激光雷达数据动态存储系统设计

Design of Dynamic Data Storage System for Massive Lidar Data Under Cloud Computing

伴随网络技术迅速发展与广泛应用,云计算逐渐成为对海量数据计算主要运算形式。由于数据通过高维形式储存,造成严重存储开销。文中以海量激光雷达数据为研究对象,在云计算下对存储系统进行优化设计。对于海量激光雷达数据,采用New-blocking分块策略,设定分块阈值包括最小分块因子值和最大分块因子值,利用阈值判定是否对文件分块,通过定义文件均匀度获得文件分块大小数据值;运用增加副本策略,根据文件热度判断其被访问情况,增加此文件block副本缓解负载。为进一步解决负载问题,运用数据动态平衡模型对激光数据进行监控、合并、拆分及数据移动等步骤处理,使物理节点通过迁移虚拟节点完成资源重分配,从而实现数据动态存储系统优化设计过程。实验证明,该系统能够有效存储激光雷达数据并达到系统资源使用率最大化的目的。

With the rapid development and wid e application of network technology,cloud computing has gradually become the main computing form of massive data computing. Because the data is stored in high dimensional form,it causes serious storage overhead. Taking the massive lidar data as the research object,the storage system is optimized under the cloud computing. The mass of laser radar data using the New-blocking block strategy,including the minimum threshold setting block block factor and the maximum block factor value,determine whether the file is divided by the threshold,by definition file uniformity get file block size data value; the increase of replication strategy,determine the visited file according to heat this file copy of block,alleviate the load increase. In order to solve the load problem,using the data of the dynamic balance model of monitoring,data integration,data separation and laser processing steps such as mobile,the physical node through the migration of virtual nodes to complete resource allocation, so as to realize the dynamic data storage system optimization design process.Experiments show that the system can effectively store lidar data and achieve the purpose of maximizing the utilization of system resources.