基于Netty和Kafka的工程机械车联网数据采集系统设计方案

Design of Data Collection System for Internet of Vehicles in Construction Machinery Based on Netty and Kafka

针对传统工程机械车联网平台在处理高并发、高吞吐量的海量数据时出现堵塞、延时的问题,设计了一种基于Netty和Kafka的高性能数据采集处理系统。首先,在数据接入方面,通过“负载均衡”进行数据分发,减少单台服务器的负载,提高系统整体的高并发、低延迟处理能力;然后充分利用Netty的NIO异步传输机制,使用Redis缓存策略,解决访问业务数据库IO开销大的问题,提高处理效率;最后,通过Kafka的异步通信机制将数据接入与业务处理进行解耦,解决因Netty线程被阻塞而导致通讯链路堵塞问题。实验证明,与传统的数据接入系统相比,该系统具备高吞吐、低延迟、易扩展等优势,能够满足工程机械车联网的数据采集实时性要求。

For the problems of blockage and delay when facing high-concurrency and high-throughput mass data in the traditional platform of internet of vehicles in construction machinery,we designed a high-performance data collection and processing system based on Netty and kafka technology.Firstly,in terms of data access,data is distributed through“load balancing”to reduce the load of a single server and improve the overall high concurrency and low latency processing capabilities of the system.Then,to solve the problem of high IO,and improve the data processing efficiency when accessing business database,we make full use of Netty NIO asynchronous transmission mechanism and the Redis cache strategy.Finally,the asynchronous communication mechanism of Kafka decouples data access from business processing,and solves the problem of blocking the communication link caused by the blocking of the Netty thread.Experiments have proved that compared with the traditional data access system,the system has the advantages of high throughput,low latency,easy expansion,etc.It can meet the real-time data collection requirements of Internet of Vehicles in construction machinery.