Towards efficient deep neural network training by FPGA-based batch-level parallelism

Training deep neural networks(DNNs)requires a significant amount of time and resources to obtain acceptable results,which severely limits its deployment in resource-limited platforms.This paper proposes DarkFPGA,a novel customizable framework to efficiently accelerate the entire DNN training on a single FPGA platform.First,we explore batch-level parallelism to enable efficient FPGA-based DNN training.Second,we devise a novel hardware architecture optimised by a batch-oriented data pattern and tiling techniques to effectively exploit parallelism.Moreover,an analytical model is developed to determine the optimal design parameters for the DarkFPGA accelerator with respect to a specific network specification and FPGA resource constraints.Our results show that the accelerator is able to perform about 10 times faster than CPU training and about a third of the energy consumption than GPU training using 8-bit integers for training VGG-like networks on the CIFAR dataset for the Maxeler MAX5 platform.

WiFi CSI Gesture Recognition Based on Parallel LSTM-FCN Deep Space-Time Neural Network

In this study,we developed a system based on deep space–time neural networks for gesture recognition.When users change or the number of gesture categories increases,the accuracy of gesture recognition decreases considerably because most gesture recognition systems cannot accommodate both user differentiation and gesture diversity.To overcome the limitations of existing methods,we designed a onedimensional parallel long short-term memory–fully convolutional network(LSTM–FCN)model to extract gesture features of different dimensions.LSTM can learn complex time dynamic information,whereas FCN can predict gestures efficiently by extracting the deep,abstract features of gestures in the spatial dimension.In the experiment,50 types of gestures of five users were collected and evaluated.The experimental results demonstrate the effectiveness of this system and robustness to various gestures and individual changes.Statistical analysis of the recognition results indicated that an average accuracy of approximately 98.9% was achieved.

Neighborhood fusion-based hierarchical parallel feature pyramid network for object detection

In order to improve the detection accuracy of small objects,a neighborhood fusion-based hierarchical parallel feature pyramid network(NFPN)is proposed.Unlike the layer-by-layer structure adopted in the feature pyramid network(FPN)and deconvolutional single shot detector(DSSD),where the bottom layer of the feature pyramid network relies on the top layer,NFPN builds the feature pyramid network with no connections between the upper and lower layers.That is,it only fuses shallow features on similar scales.NFPN is highly portable and can be embedded in many models to further boost performance.Extensive experiments on PASCAL VOC 2007,2012,and COCO datasets demonstrate that the NFPN-based SSD without intricate tricks can exceed the DSSD model in terms of detection accuracy and inference speed,especially for small objects,e.g.,4%to 5%higher mAP(mean average precision)than SSD,and 2%to 3%higher mAP than DSSD.On VOC 2007 test set,the NFPN-based SSD with 300×300 input reaches 79.4%mAP at 34.6 frame/s,and the mAP can raise to 82.9%after using the multi-scale testing strategy.

并联卷积神经网络的近红外光谱定量分析模型

近红外光谱分析已成为工农业生产过程质量监控领域中不可或缺的重要分析手段之一,在食品、农业、医药等定性定量分析领域被广泛应用。预测精度高、运行速度快、泛化能力强的近红外光谱预测模型可用于不同物质的定性定量分析。但由于近红外光谱数据量的激增,传统的近红外光谱建模方法已经出现明显的不足。随着人工智能技术的不断发展,深度学习算法在近红外光谱分析领域得到了广泛应用。提出了一种基于并联卷积神经网络的近红外光谱定量分析模型(PaBATunNet)。该模型由1个一维卷积层、1个并联卷积模块(Module)、1个展平层、4个全连接层和1个参数调节器(PR)组成,Module模块包括5个子模块分别对光谱数据进行线性及非线性多维特征提取,并通过Concatenate函数将提取后的光谱特征数据进行拼接,PR模块通过调节优化PaBATunNet模型参数,提高模型预测精度。基于Gard-CAM思想给出了PaBATunNet模型高贡献度特征波长,增加了PaBATunNet模型的可解释性。以谷物、柴油、啤酒、牛奶四组公开的近红外光谱数据为例,将PaBATunNet模型的预测结果与偏最小二乘(PLS)、主成分回归(PCR)、支持向量机(SVM)和BP神经网络(BP)模型的预测结果进行比较。结果表明,与PLS相比,PaBATunNet模型在谷物、柴油、啤酒、牛奶数据集的预测精度上分别提高了30.0%、40.7%、43.0%、52.8%;与PCR相比,PaBATunNet模型的预测精度分别提高了28.8%、35.9%、40.8%、52.2%;与SVM相比,PaBATunNet模型的预测精度分别提高了45.5%、37.4%、45.3%、54.7%;与BP相比,PaBATunNet模型的预测精度分别提高了7.9%、32.4%、90.1%、62.0%。基于并联卷积神经网络的近红外光谱建模方法相比于传统建模方法解决了模型预测精度低、运行时间长、泛化能力差以及可解释性不强等问题,可有效应用于工农业生产中不同物质的定量分析,为建立快速、无损、高精度的近红外光谱定量分析模型提供了科学基础。

深度神经网络动态分层梯度稀疏化及梯度合并优化方法

针对数据并行方法加速大规模深度神经网络时易出现的通信开销大、训练耗时长、资源利用率不高的问题,提出了一种深度神经网络动态分层梯度稀疏化及梯度合并优化方法。首先,将梯度稀疏化压缩与流水线并行技术相结合,提出动态分层梯度稀疏优化方法,为每层神经网络匹配一个合适的阈值,通过在后续迭代时动态调整该阈值,实现对每层网络传输梯度的自适应压缩。然后,提出了层梯度合并方法,利用动态规划算法对层梯度合并时的通信开销、稀疏化及层梯度计算时间进行权衡优化,求解出最佳的层梯度合并组合,并将多层小尺度梯度张量合并为一层通信,以降低分层梯度决策时引入的过高通信延迟开销。最后,将求解出的最佳层梯度合并组合应用于具体的训练迭代过程。实验结果表明:与已有方法相比,所提方法可在保证模型训练精度的同时大大降低通信开销,提升模型的训练速度;与未压缩方法相比,训练速度最大可提升1.99倍。

结合空洞卷积与注意力机制的道路提取方法

针对高分辨率影像中道路情况复杂,存在细小道路和被建筑、阴影等隔断道路,导致道路提取精度不高的问题,提出一种结合空洞卷积单元和并行注意力机制模块的改进模型AP-LinkNet。该模型是通过在下采样编码过程中扩大感受野和深层次关注道路特征以达到更高的细节道路提取精度。其中空洞卷积模块在扩大感受野的同时不改变空间上像素之间的关系,并行注意力机制提高输入影像采样过程中对通道和空间信息的关注度,并加权赋值给解码步骤的反卷积特征。结合两种机制的特点,减少复杂道路背景的噪声扰乱性以及提高道路提取模型的整体精度。与DeepLabV3+、U-Net、LinkNet和D-LinkNet模型做对比分析,AP-LinkNet模型在DeepGlobe数据集上道路提取的F_(1)分数和IOU评价指标为80.69%和78.65%,其中F_(1)分数分别高出对比模型11.71%、5.24%、3.97%和3.58%。结果表明模型精确度和鲁棒性更高,对于高分影像狭窄、被遮挡等复杂道路细节提取效果好。

基于CNN-GRU并联网络的海上风电支撑结构损伤识别

利用振动响应和深度学习进行结构损伤识别时,会遇到需要较多测点数据、损伤识别准确率不高以及网络容易发生过拟合等问题。为此,提出了一种基于卷积神经网络-门控循环单元(convolutional neural networks-gated recurrent unit,CNN-GRU)神经网络并联网络的结构损伤识别新方法。首先,对响应信号进行广义S变换(generalized S-transform,GST)得到其时频图像。然后,分别利用CNN和GRU从时频图像和响应信号中提取时频域特征和时序特征,并将时频域特征和时序特征拼接后输入全连接层和Softmax分类器中进行结构损伤识别。位移激励下的海上风电支撑结构模型试验数据验证结果表明,该方法仅需要一个测点的响应信号,与其他同类方法相比具有更高的识别准确率和效率。

多时态BIT遥感图像建筑物的变化检测

针对来自相同地理空间的不同时刻遥感图像之间的季节性和光度变化(色差)等因素所引起的干扰,提出了多时态-BIT遥感图像变化检测方法.该方法引入了过去多个不同时刻的遥感图像,融合当前遥感图像与过去时态遥感图像两两变化检测的结果,该方法有助于排除季节性和光度变化引起的误报,提高了变化检测的准确性;并且利用过去多个不同时刻的遥感图像,进一步消除非目标建筑变化的影响,其变化点像素差值引入作为损失函数正则化项,从而进一步提高变化检测的鲁棒性和可靠性.本文以三时态(3个不同时刻的遥感图像)为例,使用了遥感图像建筑物变化数据集进行了实验.实验结果表明,多时态-BIT方法相对于仅考虑两个时态的变化检测方法,在遥感图像建筑物变化检测任务中表现出更好的效果.

深度神经网络模型任务切分及并行优化方法

为解决传统手工切分神经网络模型计算任务并行化方法面临的并行化难度大、训练耗时长、设备利用率低等问题,提出了一种基于深度神经网络(DNN)模型特性感知的任务切分及并行优化方法。结合硬件计算环境,对模型计算特性进行动态分析,获取模型内部相关性和各类参数属性,构建原始计算任务有向无环图(DAG);利用增强反链,构建DAG节点间可分区聚类的拓扑关系,将原始DAG转换为易于切分的反链DAG;通过拓扑排序生成反链DAG状态序列,并使用动态规划将状态序列切分为不同执行阶段,分析最佳分割点进行模型切分,实现模型分区与各GPU间动态匹配;对批量进行微处理,通过引入流水线并行实现多迭代密集训练,提高GPU利用率,减少训练耗时。实验结果表明:与已有模型切分方法相比,在CIFAR-10数据集上,所提模型切分及并行优化方法可实现各GPU间训练任务负载均衡,在保证模型训练精度的同时,4 GPU加速比达到3.4,8 GPU加速比为3.76。

自优化双模态多通路非深度前庭神经鞘瘤识别模型

针对不同模态间对应特征极易融合错位、识别模型专家主观经验式调参且计算成本高等问题,提出自优化双模态(“对比增强T1加权”与“高分辨率增强T2加权”)多通路非深度前庭神经鞘瘤识别模型。首先,通过构建前庭神经鞘瘤识别模型进一步挖掘前庭神经鞘瘤病症多模态影像特征及模态间复杂的非线性互补信息;其次,设计基于博弈论全局并行麻雀搜索算法的模型优化策略,实现模型关键超参数的自适应寻优,使模型具有较优的识别效果。实验结果表明,相较于基于深度学习的模型,所提模型在识别准确率提升4.19个百分点的情况下参数量降低了27.9%,验证了它的有效性和自适应性。

PDP: Parallel Dynamic Programming

Deep reinforcement learning is a focus research area in artificial intelligence. The principle of optimality in dynamic programming is a key to the success of reinforcement learning methods. The principle of adaptive dynamic programming ADP is first presented instead of direct dynamic programming DP , and the inherent relationship between ADP and deep reinforcement learning is developed. Next, analytics intelligence, as the necessary requirement, for the real reinforcement learning, is discussed. Finally, the principle of the parallel dynamic programming, which integrates dynamic programming and analytics intelligence, is presented as the future computational intelligence. © 2014 Chinese Association of Automation.

Exploring compression and parallelization techniques for distribution of deep neural networks over Edge-Fog continuum-a review

Purpose-The trend of“Deep Learning for Internet of Things(IoT)”has gained fresh momentum with enormous upcoming applications employing these models as their processing engine and Cloud as their resource giant.But this picture leads to underutilization of ever-increasing device pool of IoT that has already passed 15 billion mark in 2015.Thus,it is high time to explore a different approach to tackle this issue,keeping in view the characteristics and needs of the two fields.Processing at the Edge can boost applications with realtime deadlines while complementing security.Design/methodology/approach-This review paper contributes towards three cardinal directions of research in the field of DL for IoT.The first section covers the categories of IoT devices and how Fog can aid in overcoming the underutilization of millions of devices,forming the realm of the things for IoT.The second direction handles the issue of immense computational requirements of DL models by uncovering specific compression techniques.An appropriate combination of these techniques,including regularization,quantization,and pruning,can aid in building an effective compression pipeline for establishing DL models for IoT use-cases.The third direction incorporates both these views and introduces a novel approach of parallelization for setting up a distributed systems view of DL for IoT.Findings-DL models are growing deeper with every passing year.Well-coordinated distributed execution of such models using Fog displays a promising future for the IoT application realm.It is realized that a vertically partitioned compressed deep model can handle the trade-off between size,accuracy,communication overhead,bandwidth utilization,and latency but at the expense of an additionally considerable memory footprint.To reduce the memory budget,we propose to exploit Hashed Nets as potentially favorable candidates for distributed frameworks.However,the critical point between accuracy and size for such models needs further investigation.Originality/value-To the best of our knowledge,no study has explored the inherent parallelism in deep neural network architectures for their efficient distribution over the Edge-Fog continuum.Besides covering techniques and frameworks that have tried to bring inference to the Edge,the review uncovers significant issues and possible future directions for endorsing deep models as processing engines for real-time IoT.The study is directed to both researchers and industrialists to take on various applications to the Edge for better user experience.

HOPE:a heterogeneity-oriented parallel execution engine for inference on mobiles

It is significant to efficiently support artificial intelligence(AI)applications on heterogeneous mobile platforms,especially coordinately execute a deep neural network(DNN)model on multiple computing devices of one mobile platform.This paper proposes HOPE,an end-to-end heterogeneous inference framework running on mobile platforms to distribute the operators in a DNN model to different computing devices.The problem is formalized into an integer linear programming(ILP)problem and a heuristic algorithm is proposed to determine the near-optimal heterogeneous execution plan.The experimental results demonstrate that HOPE can reduce up to 36.2%inference latency(with an average of 22.0%)than MOSAIC,22.0%(with an average of 10.2%)than StarPU and 41.8%(with an average of 18.4%)thanμLayer respectively.

嵌入式异构智能计算系统并行多流水线设计

嵌入式智能计算系统因其功耗受限和多传感器实时智能处理需要,对硬件平台的智能算力能效比和智能计算业务并行度提出了严峻挑战.传统嵌入式计算系统常采用的DSP+FPGA数字信号处理架构,无法适用于多个神经网络模型加速场景.本文基于ARM+DLP+SRIO嵌入式异构智能计算架构,利用智能处理器多片多核多内存通道特性,提出了并行多流水线设计方法.该方法充分考虑智能计算业务中数据传输、拷贝、推理、结果反馈等环节时间开销,为不同的神经网络模型合理分配智能算力资源,以达到最大的端到端智能计算业务吞吐率.实验结果表明,采用并行多流水线设计方法的深度学习处理器利用率较单流水线平均提高约25.2%,较无流水线平均提高约30.7%,满足可见光、红外、SAR等多模图像实时智能处理需求,具有实际应用价值.

基于多模型并行融合网络的恶意流量检测方法

针对单一串行深度学习检测模型提取流量特征时无法完整反映原始流量信息,且恶意流量识别精度低的问题,设计多模型并行融合网络,提出一种基于多模型并行融合网络的恶意流量检测方法。所提方法采用并行方式,融合一维卷积神经网络(1D-CNN)与双向长短期记忆(Bi-LSTM)网络进行特征提取和流量识别,各条支路均直接面向原始流量,同时提取流量的空间特征与时序特征,采用共同的全连接层进行特征融合,可更精准地反映原始流量信息并有效提高恶意流量的识别准确率。在开源NSL-KDD数据集上的实验结果表明,所提方法恶意流量检测的特征提取能力、鲁棒性以及在线学习能力等方面均表现了优越的性能。

深度学习图像重建算法在膝关节加速MRI中的临床应用研究

目的 评估使用和不使用深度学习重建(deep learning reconstruction,DLR)算法的膝关节加速二维(two dimensional,2D)快速自旋回波(fast spin echo,FSE)序列的图像质量和诊断效能。材料与方法前瞻性纳入92名怀疑有膝关节病变的患者,采用3.0 T MRI并行采集(parallel imaging,PI)基于K空间域重建(autocalibrating reconstruction for Cartesian sampling,ARC)算法进行膝关节加速2D FSE序列扫描,设置加速因子为2.0。扫描结束后系统自动保存为不使用DLR的原始图像(original images of FSE,FSE_O)和使用DLR后的FSE(deep learning reconstruction images of FSE,FSE_(DL))两组图像。采用主观(李克特5分量表,内容包括图像的整体质量、清晰度、诊断置信度)与客观定量测定图像信噪比(signal-to-noise ratio,SNR)与对比噪声比(contrast-to-noise ratio,CNR)相结合的方法对两组图像质量进行综合评价。分别测量比较膝关节质子密度加权成像(proton density weighted imaging,PDWI)、T1WI矢状位股骨下端骨髓腔、软骨、滑膜液、髌下脂肪垫、前交叉韧带各组织的SNR和软骨/滑膜液CNR。基于两组图像分别对膝关节结构异常进行评分,同时评估观察者间和观察者内评分一致性。结果四个临床标准方位加速2D FSE序列的MRI采集时间为4 min 39 s。FSE_(DL)的图像整体质量、清晰度及诊断置信度评分均高于FSE_O,其中对FSE_(DL)、FSE_O的图像清晰度评分差异有统计学意义(P<0.05)。两名医师对图像质量主观评价的一致性组内相关系数在0.710~0.898之间。使用DLR的PDWI、T1WI(PDWI_(DL)、T1WI_(DL))图像上股骨外侧髁、股骨外侧髁软骨、滑膜液、髌下脂肪垫SNR明显高于不使用DLR的PDWI、T1WI原始图像(PDWI_O、T1WI_O),PDWI_(DL)图像上软骨/滑膜液CNR明显高于PDWI_O,差异均具有统计学意义(P<0.05)。两名医师分别基于FSE_O及FSE_(DL)对膝关节结构异常进行评分,具有极好的一致性,κ值在0.954~1.000之间。比较同一名医师对两组图像的诊断结果,发现其对软骨缺损的检测和评估具有较好的一致性,κ值分别为0.769和0.771。对半月板、韧带、骨髓及滑膜液的检测和评估,诊断具有极好的一致性,κ值在0.944~1.000之间,FSE_(DL)、FSE_O对上述结构异常的检测无临床相关性差异。结论 DLR可用于膝关节PI ARC技术,在提高图像质量、保证临床诊断效能的同时5 min内完成图像采集,适用于临床各种膝关节疾病患者。

基于并行卷积神经网络和特征融合的小样本轴承故障诊断方法

在风力发电机轴承故障诊断过程中,基于深度学习的故障诊断方法受限于有限的标注样本,存在模型收敛困难和识别准确率较低等问题,为此,提出了一种基于并行卷积神经网络(P-CNN)和特征融合的小样本风机轴承故障诊断方法。首先,采用集合经验模态分解(EEMD)方法,将轴承的原始振动信号分解为若干个本征模态函数(IMF)分量以及残余分量;然后,分别对其进行了短时傅里叶变换(STFT),将其转换为时频特征图,同时构建了多个相同的卷积神经网络分支,以此作为特征提取器;最后,在融合层中,将提取到的时频域特征进行了通道特征融合,作为最终分类器的输入数据,对风机轴承进行了故障识别;并采用美国凯斯西储大学不同大小的轴承数据集,对该方法的适用性和有效性进行了验证。研究结果表明:在仅含有160个样本时,基于并行卷积神经网络(P-CNN)和特征融合的诊断方法的平均准确率高达94.5%;与支持向量机(SVM)、故障网络(FaultNet)、第一层宽卷积核深度卷积神经网络(WDCNN)相比,该诊断方法具有更高的准确率和更强的鲁棒性。

基于小波变换和混合深度学习的短期光伏功率预测

准确的短期光伏功率预测对于保证电能质量及提高电力系统运行可靠性具有重要意义。为此,文章提出了一种基于小波变换和混合深度学习的短期光伏功率预测方法。首先,将天气类型分为理想天气(晴天)和非理想天气(多云、阴天等)。对于理想天气,将历史光伏功率时间序列转化为二维图像作为混合深度学习模型(Hybrid Deep Learning Model,HDLM)的输入。对于非理想天气,使用小波变换对历史光伏功率时间序列进行分解,将得到的分量和气象参数转化成三维图像作为HDLM的输入。在HDLM中引入并行结构,由多个并列卷积神经网络和双向长短期记忆网络组成。实验结果表明,在理想天气和非理想天气条件下,所提短期光伏功率预测方法均具有较高的预测精度。

冷连轧轧制力深度神经网络模型泛化能力并行优化

为了更好调控冷连轧板厚参数,设计了一种冷连轧轧制力深度神经网络模型,增强了冷连轧模型的控制效果。选择2030冷连轧结构进行研究,对多输入多输出(MIMO)深度神经网络(DNN)进行预处理,针对多线程CPU与GPU实施了优化,对比了神经网络模型和冷连轧系统Siemens模型误差。研究结果表明:L-M算法表现出了更优的收敛稳定性、测试和验证性能、梯度下降趋势,并且收敛速度也更快。以随机方式选择200个数据并测定泛化性能测试得到,L-M算法获得了比SCG算法更大的相关系数。都是随着隐含层数的增加,获得了性能更优的神经网络模型,并且都会增加训练时间。从各项模型指标分析,L-M算法都比SCG算法的性能更优。构建神经网络轧制力模型总共包含二个隐含层、节点数介于17~30、通过L-M算法进行训练。采用神经网络轧制力模型得到的结果与实测值之间的误差比Siemens机理模型和测试值的误差更低。

基于残差连接的并行网络去噪

针对大部分的深度卷积神经网络存在着层次难以达到很深以及神经网络的深层次性能退化问题,论文基于前馈去噪卷积神经网络的模型,提出一种结合批量重整化去噪网络中的并行网络。新方法是将原有的网络层一分为二,通过增加网络的宽度而不是深度深层结构达到获取更多特征的目的。再使用残差学习和批量归一化的方法,以改善去噪质量,并且加速训练。结果表明,相比于当前比较成熟的去噪算法,论文提出的残差连接的并行网络去噪新算法的去噪效果更为突出,去噪耗时也大大降低。