基于多GPU的并行BP算法及优化

Parallel BP Algorithm and Optimization Based on Multiple GPU

在语音识别领域,基于深度神经网络(Deep Neural Network,DNN)的声学模型与传统的基于高斯混合模型的声学模型相比具有更出色的识别效果.DNN模型主要使用误差反向传播(Back Propagation,BP)算法进行训练.由于DNN参数规模非常庞大,使用BP算法对其训练是一个异常耗时的过程.目前主要通过高性能的GPU设备对DNN模型的训练训练进行加速.本文探究了BP训练算法在多GPU设备上并行化时所存在的收敛性和带宽问题,并结合NVIDIA Kepler架构的硬件特性提出一种并行优化策略.实验结果表明优化后的算法可以有效增加小mini-batch下的GPU利用率,同时减少数据传输的开销.在相同的mini-batch尺寸下,优化后的算法在4块NVIDIA Tesla K20m设备上与单个NVIDIA Tesla K20m设备相比可以取得高达3.89倍的加速比.

In the field of speech recognition,deep neural network（ DNN） has achieved a remarkable result compared with conventional GHH-HM M.The training of DNN model using backward propagation（ BP） algorithm costs immense time due to the huge network model.It has been a state of art method to accelerate BP training on modern GPU device.This paper investigates the problems faced with parallel BP training on multiple GPU devices,and proposes a parallel optimizing strategy using the hardware features provided by NVIDIA Kepler architecture.Experimental results showthat the optimized algorithm can effectively increase the utilization of GPU on small mini-batch,while reducing the overhead of data transmission.At the same mini-batch size,the optimized algorithm can achieve a3.89 times speedup on 4 NVIDIA Tesla K20 m devices than the single one.