基于Winsock的分布式数据处理技术

Distributed Data Processing Technology Based on Winsock

针对地理信息数据量快速增长带来的处理效率问题,从Winsock通信技术着手,提出了一种基于Winsock的分布式数据处理技术。从程序架构、通信模块、处理模块、可扩展性等方面设计灵活的C/S程序,将局域网中多台计算机组建成工作群,以服务器端发布任务指令、多个客户端接收并执行任务指令的方式进行分布式数据处理,从而快速提升数据处理效率;然后以常用的遥感影像处理NDVI和NDWI计算为例,采用GDAL开源库开发数据处理模块,模拟单机作业和多机分布式协同作业进行数据处理效率测试。实验结果表明,采用分布式协同作业方式的数据处理效率明显提升。该项技术的开发难度低,且具有较好的可扩展性,可为今后大数据量数据处理提供新思路。

Aiming at the processing efficiency problem caused by the geographical information data rapid growth,starting with Winsock communication technology,we proposed a distributed data processing technology based on Winsock.We designed flexible C/S programs from the aspects of program architecture,communication module,processing module and scalability,and formed multiple computers in LAN into working groups.We carried out the distributed data processing by issuing task instructions on the server,and receiving and executing task instructions on multiple clients,thus rapidly improving data processing efficiency.Then,taking the common remote sensing image processing NDVI and NDWI calculation for example,we used GDAL open source library to develop the data processing module,simulated the single machine operation and multi-machine distributed cooperative operation,and tested the data processing efficiency.The experimental result shows that the data processing efficiency is significantly improved by using the distributed cooperative operation.This technology has low development difficulty and good scalability,which can provide a new idea for large amount of data processing in the future.