面向股票交易分析场景的流式大数据系统测试框架

System Test Framework of Stream Data for Stock Trading Analysis Scenario

分布式集群环境使得数据实时计算更为复杂,流式大数据处理系统的正确性难以保障.现有的大数据基准测试框架可以测试流式大数据处理系统的性能表现,但是普遍存在应用场景设计简单、评价指标不充分等不足.针对这一挑战,本文构造了一个面向股票交易场景的流式大数据基准测试框架,通过生成股票高频交易数据,测试系统在高流速场景下的延迟、吞吐量、GC时间、CPU资源等的性能表现.同时,通过横向测试验证流式大数据系统的扩展性.本文以Apache Spark Streaming为待测系统进行测试,实验结果表明,高流速场景下出现延迟增加、GC时间提高等性能下降问题,原因是系统输入速率的提高及并行度的增加.

Distributed cluster environment makes real-time data computation more complex, and the correctness of stream large data processing system is difficult to guarantee. The existing large data benchmarking framework can test the performance of stream large data processing system, but there are many shortcomings such as simple application scenario design and inadequate evaluation index. To address this challenge, this study constructs a stream large data benchmarking framework for stock trading scenarios, generates high-frequency stock trading data through a flow-based data generator, and tests the performance of the system in high-speed scenarios in terms of delay, throughput, GC time, CPU resources, and so on. At the same time, the scalability of large data streaming system is verified by horizontal test. In this study, Apache Spark Streaming is used as the test system to test. The experimental results show that the performance degradation problems such as delay increase and GC time increase occur in high-speed scenarios because of the increase of input rate and parallelism of the system.