基于迁移学习的并行化大数据流传输系统设计

Design of parallel big data stream transmission system based on transfer learning

传统系统在并行化大数据流组件不变情况下的吞吐量会随着并发数增多而减少,影响数据流传输效率。为了解决这一问题,提出基于迁移学习的并行化大数据流传输系统。系统硬件由FPGA核心控制器、XC7K325T-2FFG 900芯片、DCM时钟组成,用于实时传输数据流。系统软件是在STORM平台上引入迁移学习算法。软、硬件结合,完成基于迁移学习的并行化大数据流传输系统设计。实验分别测试了两个系统在并行化大数据流组件不变情况下的吞吐量。将并行化大数据流分类组件设置为(5.5),(5.6),(5.7),(5.8),从实验结果可知,所设计的系统吞吐量会随着并发数、线程增多,呈上升趋势,能够有效提升并行化大数据流传输效率。

As the throughput of the traditional system could decrease with the increase of the concurrent amount under the circumstance that the parallel big data stream component is unchanged,which will affect the transmission efficiency of data stream,a parallel big data stream transmission system based on transfer learning is proposed.The system hardware is composed of FPGA core controller,XC7K325T⁃2FFG900 chip and DCM clock,which is used to transmit data stream in real time.The system software can realize the introduction of transfer learning algorithm on the STORM platform.The design of the parallel big data stream transmission system based on transfer learning is completed in combination of the software and the hardware.In the experiment,the throughputs of the two systems under the circumstance that the parallel big data flow components are unchanged are tested.The parallel big data stream classification components are set to(5.5),(5.6),(5.7)and(5.8).It can be seen from the experimental results that the throughput of the designed system can be on the rise with the increase of the concurrent amount and threads,which can effectively improve the transmission efficiency of parallel big data stream.