一种利用并发提高数据处理吞吐率的模型

Real-Time and Large Throughput Data Processing System Using Concurrency

针对分布式组网终端集中式数据采集与处理系统实时性高、数据吞吐量大、周期性强等特点,引入了一种利用并发机制的大吞吐量实时数据处理模型,在讨论模型基本结构和原理的基础上,深入研究了决定模型数据吞吐效率和时效性能的并发缓存机制和算法设计。利用该模型实现的数据采集处理系统在基于GPRS组网的1500个自动气象站数据采集与处理业务应用中,性能稳定可靠,能够满足对周期性大批量实时"浪涌"数据进行实时快速处理的要求,大大提高了数据处理的吞吐效率。

Data acquiring and processing from distributed terminal unit by network are usually real-time,large throughput and periodic,so data acquisition,processing and result should be done within data output period in high-efficiency.A real-time and large throughput data processing model using concurrency is built for this given application.The working principle and basic structure for the model are discussed in detail,also concurrency for the model and corresponding key buffer which concurrent threads put data into and gets data out are analyzed in particular.As is used in observation application networked by 1500 automatic weather stations,which has indicated that the model is capable in processing periodic surge observation data and has higher data throughput,as well as implements high speed and accurate results,high stability and reliability.