A Survey of Accelerator Architectures for Deep Neural Networks

Recently,due to the availability of big data and the rapid growth of computing power,artificial intelligence(AI)has regained tremendous attention and investment.Machine learning(ML)approaches have been successfully applied to solve many problems in academia and in industry.Although the explosion of big data applications is driving the development of ML,it also imposes severe challenges of data processing speed and scalability on conventional computer systems.Computing platforms that are dedicatedly designed for AI applications have been considered,ranging from a complement to von Neumann platforms to a“must-have”and stand-alone technical solution.These platforms,which belong to a larger category named“domain-specific computing,”focus on specific customization for AI.In this article,we focus on summarizing the recent advances in accelerator designs for deep neural networks(DNNs)-that is,DNN accelerators.We discuss various architectures that support DNN executions in terms of computing units,dataflow optimization,targeted network topologies,architectures on emerging technologies,and accelerators for emerging applications.We also provide our visions on the future trend of AI chip designs.

Architecture mode, sedimentary evolution and controlling factors of deepwater turbidity channels： A case study of the M Oilfield in West Africa

Turbidity channels have been considered as one of the important types of deepwater reservoir, and the study of their architecture plays a key role in efficient development of an oil field. To better understand the reservoir architecture of the lower Congo Basin M oilfield, semiquantitative–quantitative study on turbidity channel depositional architecture patterns in the middle to lower slopes was conducted with the aid of abundant high quality materials（core, outcrop, logging and seismic data）,employing seismic stratigraphy, seismic sedimentology and sedimentary petrography methods. Then, its sedimentary evolution was analyzed accordingly. The results indicated that in the study area, grade 3 to grade 5 architecture units were single channel, complex channel and channel systems, respectively. Single channel sinuosity is negatively correlated with the slope, as internal grains became finer and thickness became thinner from bottom to top, axis to edge. The migration type of a single channel within one complex channel can be lateral migration and along paleocurrent migration horizontally, and lateral,indented and swing stacking in section view. Based on external morphological characteristics and boundaries,channel systems are comprised of a weakly confining type and a non-confining type. The O73 channel system can be divided into four complex channels named S1–S4, from bottom to top, with gradually less incision and more accretion. The study in this article will promote deeper understanding of turbidity channel theory, guide 3D geological modeling in reservoir development and contribute to efficient development of such reservoirs.

Scale adaptive fitness evaluation‐based particle swarm optimisation for hyperparameter and architecture optimisation in neural networks and deep learning

Research into automatically searching for an optimal neural network(NN)by optimi-sation algorithms is a significant research topic in deep learning and artificial intelligence.However,this is still challenging due to two issues:Both the hyperparameter and ar-chitecture should be optimised and the optimisation process is computationally expen-sive.To tackle these two issues,this paper focusses on solving the hyperparameter and architecture optimization problem for the NN and proposes a novel light‐weight scale‐adaptive fitness evaluation‐based particle swarm optimisation(SAFE‐PSO)approach.Firstly,the SAFE‐PSO algorithm considers the hyperparameters and architectures together in the optimisation problem and therefore can find their optimal combination for the globally best NN.Secondly,the computational cost can be reduced by using multi‐scale accuracy evaluation methods to evaluate candidates.Thirdly,a stagnation‐based switch strategy is proposed to adaptively switch different evaluation methods to better balance the search performance and computational cost.The SAFE‐PSO algorithm is tested on two widely used datasets:The 10‐category(i.e.,CIFAR10)and the 100−cate-gory(i.e.,CIFAR100).The experimental results show that SAFE‐PSO is very effective and efficient,which can not only find a promising NN automatically but also find a better NN than compared algorithms at the same computational cost.

Deep reinforcement learning based multi-level dynamic reconfiguration for urban distribution network:a cloud-edge collaboration architecture

With the construction of the power Internet of Things(IoT),communication between smart devices in urban distribution networks has been gradually moving towards high speed,high compatibility,and low latency,which provides reliable support for reconfiguration optimization in urban distribution networks.Thus,this study proposed a deep reinforcement learning based multi-level dynamic reconfiguration method for urban distribution networks in a cloud-edge collaboration architecture to obtain a real-time optimal multi-level dynamic reconfiguration solution.First,the multi-level dynamic reconfiguration method was discussed,which included feeder-,transformer-,and substation-levels.Subsequently,the multi-agent system was combined with the cloud-edge collaboration architecture to build a deep reinforcement learning model for multi-level dynamic reconfiguration in an urban distribution network.The cloud-edge collaboration architecture can effectively support the multi-agent system to conduct“centralized training and decentralized execution”operation modes and improve the learning efficiency of the model.Thereafter,for a multi-agent system,this study adopted a combination of offline and online learning to endow the model with the ability to realize automatic optimization and updation of the strategy.In the offline learning phase,a Q-learning-based multi-agent conservative Q-learning(MACQL)algorithm was proposed to stabilize the learning results and reduce the risk of the next online learning phase.In the online learning phase,a multi-agent deep deterministic policy gradient(MADDPG)algorithm based on policy gradients was proposed to explore the action space and update the experience pool.Finally,the effectiveness of the proposed method was verified through a simulation analysis of a real-world 445-node system.

Hybrid Deep Learning Architecture to Forecast Maximum Load Duration Using Time-of-Use Pricing Plans

Load forecasting has received crucial research attention to reduce peak load and contribute to the stability of power grid using machine learning or deep learning models.Especially,we need the adequate model to forecast the maximum load duration based on time-of-use,which is the electricity usage fare policy in order to achieve the goals such as peak load reduction in a power grid.However,the existing single machine learning or deep learning forecasting cannot easily avoid overfitting.Moreover,a majority of the ensemble or hybrid models do not achieve optimal results for forecasting the maximum load duration based on time-of-use.To overcome these limitations,we propose a hybrid deep learning architecture to forecast maximum load duration based on time-of-use.Experimental results indicate that this architecture could achieve the highest average of recall and accuracy(83.43%)compared to benchmark models.To verify the effectiveness of the architecture,another experimental result shows that energy storage system(ESS)scheme in accordance with the forecast results of the proposed model(LSTM-MATO)in the architecture could provide peak load cost savings of 17,535,700 KRW each year comparing with original peak load costs without the method.Therefore,the proposed architecture could be utilized for practical applications such as peak load reduction in the grid.

Deep Neural Network Architecture Search via Decomposition-Based Multi-Objective Stochastic Fractal Search

Deep neural networks often outperform classical machine learning algorithms in solving real-world problems.However,designing better networks usually requires domain expertise and consumes significant time and com-puting resources.Moreover,when the task changes,the original network architecture becomes outdated and requires redesigning.Thus,Neural Architecture Search(NAS)has gained attention as an effective approach to automatically generate optimal network architectures.Most NAS methods mainly focus on achieving high performance while ignoring architectural complexity.A myriad of research has revealed that network performance and structural complexity are often positively correlated.Nevertheless,complex network structures will bring enormous computing resources.To cope with this,we formulate the neural architecture search task as a multi-objective optimization problem,where an optimal architecture is learned by minimizing the classification error rate and the number of network parameters simultaneously.And then a decomposition-based multi-objective stochastic fractal search method is proposed to solve it.In view of the discrete property of the NAS problem,we discretize the stochastic fractal search step size so that the network architecture can be optimized more effectively.Additionally,two distinct update methods are employed in step size update stage to enhance the global and local search abilities adaptively.Furthermore,an information exchange mechanism between architectures is raised to accelerate the convergence process and improve the efficiency of the algorithm.Experimental studies show that the proposed algorithm has competitive performance comparable to many existing manual and automatic deep neural network generation approaches,which achieved a parameter-less and high-precision architecture with low-cost on each of the six benchmark datasets.

An Optimized Convolution Neural Network Architecture for Paddy Disease Classification

Plant disease classification based on digital pictures is challenging.Machine learning approaches and plant image categorization technologies such as deep learning have been utilized to recognize,identify,and diagnose plant diseases in the previous decade.Increasing the yield quantity and quality of rice forming is an important cause for the paddy production countries.However,some diseases that are blocking the improvement in paddy production are considered as an ominous threat.Convolution Neural Network(CNN)has shown a remarkable performance in solving the early detection of paddy leaf diseases based on its images in the fast-growing era of science and technology.Nevertheless,the significant CNN architectures construction is dependent on expertise in a neural network and domain knowledge.This approach is time-consuming,and high computational resources are mandatory.In this research,we propose a novel method based on Mutant Particle swarm optimization(MUT-PSO)Algorithms to search for an optimum CNN architecture for Paddy leaf disease classification.Experimentation results show that Mutant Particle swarm optimization Convolution Neural Network(MUTPSO-CNN)can find optimumCNNarchitecture that offers better performance than existing hand-crafted CNN architectures in terms of accuracy,precision/recall,and execution time.

Parallel Extraction of Marine Targets Applying OIDA Architecture

Computing resources are one of the key factors restricting the extraction of marine targets by using deep learning.In order to increase computing speed and shorten the computing time,parallel distributed architecture is adopted to extract marine targets.The advantages of two distributed architectures,Parameter Server and Ring-allreduce architecture,are combined to design a parallel distributed architecture suitable for deep learning–Optimal Interleaved Distributed Architecture(OIDA).Three marine target extraction methods including OTD_StErf,OTD_Loglogistic and OTD_Sgmloglog are used to test OIDA,and a total of 18 experiments in 3categories are carried out.The results show that OIDA architecture can meet the timeliness requirements of marine target extraction.The average speed of target parallel extraction with single-machine 8-core CPU is 5.75 times faster than that of single-machine single-core CPU,and the average speed with 5-machine 40-core CPU is 20.75 times faster.

Modified Visual Geometric Group Architecture for MRI Brain Image Classification

The advancement of automated medical diagnosis in biomedical engineering has become an important area of research.Image classification is one of the diagnostic approaches that do not require segmentation which can draw quicker inferences.The proposed non-invasive diagnostic support system in this study is considered as an image classification system where the given brain image is classified as normal or abnormal.The ability of deep learning allows a single model for feature extraction as well as classification whereas the rational models require separate models.One of the best models for image localization and classification is the Visual Geometric Group(VGG)model.In this study,an efficient modified VGG architecture for brain image classification is developed using transfer learning.The pooling layer is modified to enhance the classification capability of VGG architecture.Results show that the modified VGG architecture outperforms the conventional VGG architecture with a 5%improvement in classification accuracy using 16 layers on MRI images of the REpository of Molecular BRAin Neoplasia DaTa(REMBRANDT)database.

基于Deep Belief Nets的中文名实体关系抽取

关系抽取是信息抽取的一项子任务,用以识别文本中实体之间的语义关系.提出一种利用DBN(deepbelief nets)模型进行基于特征的实体关系抽取方法,该模型是由多层无监督的RBM(restricted Boltzmann machine)网络和一层有监督的BP(back-propagation)网络组成的神经网络分类器.RBM网络以确保特征向量映射达到最优,最后一层BP网络分类RBM网络的输出特征向量,从而训练实体关系分类器.在ACE04语料上进行的相关测试,一方面证明了字特征比词特征更适用于中文关系抽取任务;另一方面设计了3组不同的实验,分别使用正确的实体类别信息、通过实体类型分类器得到实体类型信息和不使用实体类型信息,用以比较实体类型信息对关系抽取效果的影响.实验结果表明,DBN非常适用于基于高维空间特征的信息抽取任务,获得的效果比SVM和反向传播网络更好.

基于知识的Deep Web集成环境变化处理的研究

研究了Deep Web集成环境中构件的依赖关系(执行偏序依赖和知识依赖),并在此基础上提出了一种基于知识的环境变化的处理方法,包括Deep Web集成环境变化处理模型以及适应Deep Web环境变化的动态体系结构和处理算法,可以对大规模Deep Web集成的进一步探索和走向应用提供参考.实验结果表明,该方法不仅可以处理Deep Web环境的变化,还可以大幅度提高集成系统的性能.

AAUConvNeXt:Enhancing Crop Lodging Segmentation with Optimized Deep Learning Architectures

Rice lodging,a phenomenon precipitated by environmental factors or crop characteristics,presents a substantial challenge in agricultural production,notably impacting yield prediction and disaster assessment.Despite that the application of conventional methodologies like visual assessment,mathematical models,and satellite remote sensing technologies has been employed in the segmentation of crop lodging,these approaches are still constrained in precision,immediacy,and capacity for large-scale evaluation.

Feature-Based Aggregation and Deep Reinforcement Learning:A Survey and Some New Implementations

In this paper we discuss policy iteration methods for approximate solution of a finite-state discounted Markov decision problem, with a focus on feature-based aggregation methods and their connection with deep reinforcement learning schemes. We introduce features of the states of the original problem, and we formulate a smaller "aggregate" Markov decision problem, whose states relate to the features. We discuss properties and possible implementations of this type of aggregation, including a new approach to approximate policy iteration. In this approach the policy improvement operation combines feature-based aggregation with feature construction using deep neural networks or other calculations. We argue that the cost function of a policy may be approximated much more accurately by the nonlinear function of the features provided by aggregation, than by the linear function of the features provided by neural networkbased reinforcement learning, thereby potentially leading to more effective policy improvement.

基于DeepLabv3架构的高分辨率遥感图像分类

针对目前使用机器学习解决高分辨率遥感图像分类主要存在下采样导致的细节信息丢失问题,提出了一种基于DeepLabv3架构的小波域DeepLabv3-MRF(Markov random field,MRF)算法。选择当前较为普遍的DeepLabv3架构分类算法,能够获得更为精确的分类结果;采用小波域DeepLabv3-MRF算法,还能够获得更为清晰的边缘细节信息。选取南方某地区高分辨率无人机遥感图像进行分类实验,通过小波变换的方向性、非冗余性以及MRF变换像素空间的交互性这三个方面,将分类结果与原始DeepLabv3架构分类结果对比分析。结果表明,所提出的分类方法精度明显高于原始DeepLabv3架构分类算法的精度,总体精度可提升3%左右,并且可以充分表达高分辨率遥感图像细节信息。

A comparison of deep learning methods for seismic impedance inversion

Deep learning is widely used for seismic impedance inversion,but few work provides in-depth research and analysis on designing the architectures of deep neural networks and choosing the network hyperparameters.This paper is dedicated to comprehensively studying on the significant aspects of deep neural networks that affect the inversion results.We experimentally reveal how network hyperparameters and architectures affect the inversion performance,and develop a series of methods which are proven to be effective in reconstructing high-frequency information in the estimated impedance model.Experiments demonstrate that the proposed multi-scale architecture is helpful to reconstruct more high-frequency details than a conventional network.Besides,the reconstruction of high-frequency information can be further promoted by introducing a perceptual loss and a generative adversarial network from the computer vision perspective.More importantly,the experimental results provide valuable references for designing proper network architectures in the seismic inversion problem.

传统村落航拍图像中民族建筑识别的DeepLabV3+改进算法

针对传统村落航拍图像难以准确识别民族建筑目标的问题,本文分析了3种深度学习像素分类算法的识别结果,DeepLabV3+、U-Net、PSP-Net等算法的民族建筑识别准确率分别为0.957、0.929、0.943,利用DeepLabV3+算法对传统村落6个典型区域进行测试,测试结果存在的主要问题包括:传统村落部分道路和场地区域被标记为民族建筑,标记的民族建筑区域边缘呈锯齿状。为能够准确地识别传统村落的民族建筑,依托于ArcGIS软件平台对DeepLabV3+算法的民族建筑识别结果进行改进处理,改进处理的内容包括标记区域过滤和标记区域边界清理等。改进处理的结果能在确保民族建筑识别准确率的基础上,优化传统村落航拍图像的民族建筑标记区域边缘的平滑程度。

An Online Chronic Disease Prediction System Based on Incremental Deep Neural Network

Many chronic disease prediction methods have been proposed to predict or evaluate diabetes through artificial neural network.However,due to the complexity of the human body,there are still many challenges to face in that process.One of them is how to make the neural network prediction model continuously adapt and learn disease data of different patients,online.This paper presents a novel chronic disease prediction system based on an incremental deep neural network.The propensity of users suffering from chronic diseases can continuously be evaluated in an incremental manner.With time,the system can predict diabetes more and more accurately by processing the feedback information.Many diabetes prediction studies are based on a common dataset,the Pima Indians diabetes dataset,which has only eight input attributes.In order to determine the correlation between the pathological characteristics of diabetic patients and their daily living resources,we have established an in-depth cooperation with a hospital.A Chinese diabetes dataset with 575 diabetics was created.Users’data collected by different sensors were used to train the network model.We evaluated our system using a real-world diabetes dataset to confirm its effectiveness.The experimental results show that the proposed system can not only continuously monitor the users,but also give early warning of physiological data that may indicate future diabetic ailments.

Mammogram Learning System for Breast Cancer Diagnosis Using Deep Learning SVM

The most common form of cancer for women is breast cancer.Recent advances in medical imaging technologies increase the use of digital mammograms to diagnose breast cancer.Thus,an automated computerized system with high accuracy is needed.In this study,an efficient Deep Learning Architecture(DLA)with a Support Vector Machine(SVM)is designed for breast cancer diagnosis.It combines the ideas from DLA with SVM.The state-of-the-art Visual Geometric Group(VGG)architecture with 16 layers is employed in this study as it uses the small size of 3×3 convolution filters that reduces system complexity.The softmax layer in VGG assumes that the training samples belong to exactly only one class,which is not valid in a real situation,such as in medical image diagnosis.To overcome this situation,SVM is employed instead of the softmax layer in VGG.Data augmentation is also employed as DLA usually requires a large number of samples.VGG model with different SVM kernels is built to classify the mammograms.Results show that the VGG-SVM model has good potential for the classification of Mammographic Image Analysis Society(MIAS)database images with an accuracy of 98.67%,sensitivity of 99.32%,and specificity of 98.34%.

广义确定性标识网络

随着智能制造、智能交通等重大国家战略实施,确定性成为信息网络尤其是行业专网的新焦点.现有确定性网络技术始终关注网络传输要素(带宽、时隙等)来保障数据流的确定性传输.然而,仅靠保障传输要素无法支撑新兴行业应用的多样化需求.例如,在算网融合场景,智算任务要求同时保障传输与计算要素的确定性来实现高性能通信;在绿色通信场景,需要考虑节点能量要素的确定性以维持网络稳定运行.针对上述需求,本文基于前期提出的标识网络技术,研究面向传输、计算、存储、能量等多要素的广义确定性网络.首先提出广义确定性标识网络架构,包括差异化服务层、异构融合网络层和智慧化适配层.差异化服务层和异构融合网络层,分别实现差异化确定性应用需求和异构化确定性网络要素的统一标识和描述,并通过标识解析映射实现确定性信息向智慧化适配层的统一封装和传递;智慧化适配层完成差异化确定性应用需求和异构化确定性网络要素的适配.现有确定性资源适配方法,即使仅考虑单一网络内的基本确定性要素,仍面临计算时间长、求解复杂性高、灵活度低等问题,为了支持更加复杂的多确定性要素、多种异构网络的协同适配,设计了基于深度强化学习的端到端的确定性调度(End-to-end Deterministic resource scheduling,E2eDet)算法,该算法可统一化、端到端地为混合数据流协同分配多种确定性网络资源,满足不同应用的差异化确定性需求.实验表明,E2eDet比DeepCQF和Random算法分别提升了28.4%和6.38倍数据流调度数量,同时E2eDet可以较好地权衡计算时间和调度能力.

Deep Learning Applied to Computational Mechanics:A Comprehensive Review,State of the Art,and the Classics

Three recent breakthroughs due to AI in arts and science serve as motivation:An award winning digital image,protein folding,fast matrix multiplication.Many recent developments in artificial neural networks,particularly deep learning(DL),applied and relevant to computational mechanics(solid,fluids,finite-element technology)are reviewed in detail.Both hybrid and pure machine learning(ML)methods are discussed.Hybrid methods combine traditional PDE discretizations with ML methods either(1)to help model complex nonlinear constitutive relations,(2)to nonlinearly reduce the model order for efficient simulation(turbulence),or(3)to accelerate the simulation by predicting certain components in the traditional integration methods.Here,methods(1)and(2)relied on Long-Short-Term Memory(LSTM)architecture,with method(3)relying on convolutional neural networks.Pure ML methods to solve(nonlinear)PDEs are represented by Physics-Informed Neural network(PINN)methods,which could be combined with attention mechanism to address discontinuous solutions.Both LSTM and attention architectures,together with modern and generalized classic optimizers to include stochasticity for DL networks,are extensively reviewed.Kernel machines,including Gaussian processes,are provided to sufficient depth for more advanced works such as shallow networks with infinite width.Not only addressing experts,readers are assumed familiar with computational mechanics,but not with DL,whose concepts and applications are built up from the basics,aiming at bringing first-time learners quickly to the forefront of research.History and limitations of AI are recounted and discussed,with particular attention at pointing out misstatements or misconceptions of the classics,even in well-known references.Positioning and pointing control of a large-deformable beam is given as an example.