In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of ...In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.展开更多
甚高频数据交换系统(Very high frequency Data Exchange System,VDES)作为新一代船舶通信系统,具有广阔的应用前景。由于卫星相对船舶的高速运动,VDES中上行应用特定消息(Application-specific Message,ASM)链路会产生较大的多普勒频移...甚高频数据交换系统(Very high frequency Data Exchange System,VDES)作为新一代船舶通信系统,具有广阔的应用前景。由于卫星相对船舶的高速运动,VDES中上行应用特定消息(Application-specific Message,ASM)链路会产生较大的多普勒频移,在接收端仅依靠已知训练序列估计的频偏等信道参数无法满足正确解调的性能要求。为此提出一种基于判决反馈的解调方法,通过分段解调,缩短每次解调的数据长度,提高解调时对频偏的容忍度,并利用每段解调的结果作为下一段未解调数据的导频,估计出当前数据中的信道参数。仿真结果表明,所提算法相较于无反馈相干解调算法性能大大提升。在上述研究的基础上,在可编程逻辑器件上实现了对ASM无导频上行链路的正确解调。展开更多
基金supported by the Research Fund of National Key Laboratory of Computer Architecture under Grant No.CARCH201501the Open Project Program of the State Key Laboratory of Mathematical Engineering and Advanced Computing under Grant No.2016A09
文摘In the era of Big Data, typical architecture of distributed real-time stream processing systems is the combination of Flume, Kafka, and Storm. As a kind of distributed message system, Kafka has the characteristics of horizontal scalability and high throughput, which is manly deployed in many areas in order to address the problem of speed mismatch between message producers and consumers. When using Kafka, we need to quickly receive data sent by producers. In addition, we need to send data to consumers quickly. Therefore, the performance of Kafka is of critical importance to the performance of the whole stream processing system. In this paper, we propose the improved design of real-time stream processing systems, and focus on improving the Kafka's data loading process.We use Kafka cat to transfer data from the source to Kafka topic directly, which can reduce the network transmission. We also utilize the memory file system to accelerate the process of data loading, which can address the bottleneck and performance problems caused by disk I/O. Extensive experiments are conducted to evaluate the performance, which show the superiority of our improved design.
文摘甚高频数据交换系统(Very high frequency Data Exchange System,VDES)作为新一代船舶通信系统,具有广阔的应用前景。由于卫星相对船舶的高速运动,VDES中上行应用特定消息(Application-specific Message,ASM)链路会产生较大的多普勒频移,在接收端仅依靠已知训练序列估计的频偏等信道参数无法满足正确解调的性能要求。为此提出一种基于判决反馈的解调方法,通过分段解调,缩短每次解调的数据长度,提高解调时对频偏的容忍度,并利用每段解调的结果作为下一段未解调数据的导频,估计出当前数据中的信道参数。仿真结果表明,所提算法相较于无反馈相干解调算法性能大大提升。在上述研究的基础上,在可编程逻辑器件上实现了对ASM无导频上行链路的正确解调。
