GShuttle:Optimizing Memory Access Efficiency for Graph Convolu-tional Neural Network Accelerators

Graph convolutional neural networks(GCNs)have emerged as an effective approach to extending deep learning for graph data analytics,but they are computationally challenging given the irregular graphs and the large num-ber of nodes in a graph.GCNs involve chain sparse-dense matrix multiplications with six loops,which results in a large de-sign space for GCN accelerators.Prior work on GCN acceleration either employs limited loop optimization techniques,or determines the design variables based on random sampling,which can hardly exploit data reuse efficiently,thus degrading system efficiency.To overcome this limitation,this paper proposes GShuttle,a GCN acceleration scheme that maximizes memory access efficiency to achieve high performance and energy efficiency.GShuttle systematically explores loop opti-mization techniques for GCN acceleration,and quantitatively analyzes the design objectives(e.g.,required DRAM access-es and SRAM accesses)by analytical calculation based on multiple design variables.GShuttle further employs two ap-proaches,pruned search space sweeping and greedy search,to find the optimal design variables under certain design con-straints.We demonstrated the efficacy of GShuttle by evaluation on five widely used graph datasets.The experimental simulations show that GShuttle reduces the number of DRAM accesses by a factor of 1.5 and saves energy by a factor of 1.7 compared with the state-of-the-art approaches.

Pluggable multitask diffractive neural networks based on cascaded metasurfaces

Optical neural networks have significant advantages in terms of power consumption,parallelism,and high computing speed,which has intrigued extensive attention in both academic and engineering communities.It has been considered as one of the powerful tools in promoting the fields of imaging processing and object recognition.However,the existing optical system architecture cannot be reconstructed to the realization of multi-functional artificial intelligence systems simultaneously.To push the development of this issue,we propose the pluggable diffractive neural networks(P-DNN),a general paradigm resorting to the cascaded metasurfaces,which can be applied to recognize various tasks by switching internal plug-ins.As the proof-of-principle,the recognition functions of six types of handwritten digits and six types of fashions are numerical simulated and experimental demonstrated at near-infrared regimes.Encouragingly,the proposed paradigm not only improves the flexibility of the optical neural networks but paves the new route for achieving high-speed,low-power and versatile artificial intelligence systems.

Hybrid model for BOF oxygen blowing time prediction based on oxygen balance mechanism and deep neural network

The amount of oxygen blown into the converter is one of the key parameters for the control of the converter blowing process,which directly affects the tap-to-tap time of converter. In this study, a hybrid model based on oxygen balance mechanism (OBM) and deep neural network (DNN) was established for predicting oxygen blowing time in converter. A three-step method was utilized in the hybrid model. First, the oxygen consumption volume was predicted by the OBM model and DNN model, respectively. Second, a more accurate oxygen consumption volume was obtained by integrating the OBM model and DNN model. Finally, the converter oxygen blowing time was calculated according to the oxygen consumption volume and the oxygen supply intensity of each heat. The proposed hybrid model was verified using the actual data collected from an integrated steel plant in China, and compared with multiple linear regression model, OBM model, and neural network model including extreme learning machine, back propagation neural network, and DNN. The test results indicate that the hybrid model with a network structure of 3 hidden layer layers, 32-16-8 neurons per hidden layer, and 0.1 learning rate has the best prediction accuracy and stronger generalization ability compared with other models. The predicted hit ratio of oxygen consumption volume within the error±300 m^(3)is 96.67%;determination coefficient (R^(2)) and root mean square error (RMSE) are0.6984 and 150.03 m^(3), respectively. The oxygen blow time prediction hit ratio within the error±0.6 min is 89.50%;R2and RMSE are0.9486 and 0.3592 min, respectively. As a result, the proposed model can effectively predict the oxygen consumption volume and oxygen blowing time in the converter.

Activation Redistribution Based Hybrid Asymmetric Quantization Method of Neural Networks

The demand for adopting neural networks in resource-constrained embedded devices is continuously increasing.Quantization is one of the most promising solutions to reduce computational cost and memory storage on embedded devices.In order to reduce the complexity and overhead of deploying neural networks on Integeronly hardware,most current quantization methods use a symmetric quantization mapping strategy to quantize a floating-point neural network into an integer network.However,although symmetric quantization has the advantage of easier implementation,it is sub-optimal for cases where the range could be skewed and not symmetric.This often comes at the cost of lower accuracy.This paper proposed an activation redistribution-based hybrid asymmetric quantizationmethod for neural networks.The proposedmethod takes data distribution into consideration and can resolve the contradiction between the quantization accuracy and the ease of implementation,balance the trade-off between clipping range and quantization resolution,and thus improve the accuracy of the quantized neural network.The experimental results indicate that the accuracy of the proposed method is 2.02%and 5.52%higher than the traditional symmetric quantization method for classification and detection tasks,respectively.The proposed method paves the way for computationally intensive neural network models to be deployed on devices with limited computing resources.Codes will be available on https://github.com/ycjcy/Hybrid-Asymmetric-Quantization.

A data-driven model of drop size prediction based on artificial neural networks using small-scale data sets

An artificial neural network(ANN)method is introduced to predict drop size in two kinds of pulsed columns with small-scale data sets.After training,the deviation between calculate and experimental results are 3.8%and 9.3%,respectively.Through ANN model,the influence of interfacial tension and pulsation intensity on the droplet diameter has been developed.Droplet size gradually increases with the increase of interfacial tension,and decreases with the increase of pulse intensity.It can be seen that the accuracy of ANN model in predicting droplet size outside the training set range is reach the same level as the accuracy of correlation obtained based on experiments within this range.For two kinds of columns,the drop size prediction deviations of ANN model are 9.6%and 18.5%and the deviations in correlations are 11%and 15%.

Multi-Scale-Matching neural networks for thin plate bending problem

Physics-informed neural networks are a useful machine learning method for solving differential equations,but encounter challenges in effectively learning thin boundary layers within singular perturbation problems.To resolve this issue,multi-scale-matching neural networks are proposed to solve the singular perturbation problems.Inspired by matched asymptotic expansions,the solution is decomposed into inner solutions for small scales and outer solutions for large scales,corresponding to boundary layers and outer regions,respectively.Moreover,to conform neural networks,we introduce exponential stretched variables in the boundary layers to avoid semiinfinite region problems.Numerical results for the thin plate problem validate the proposed method.

Finite-time Prescribed Performance Time-Varying Formation Control for Second-Order Multi-Agent Systems With Non-Strict Feedback Based on a Neural Network Observer

This paper studies the problem of time-varying formation control with finite-time prescribed performance for nonstrict feedback second-order multi-agent systems with unmeasured states and unknown nonlinearities.To eliminate nonlinearities,neural networks are applied to approximate the inherent dynamics of the system.In addition,due to the limitations of the actual working conditions,each follower agent can only obtain the locally measurable partial state information of the leader agent.To address this problem,a neural network state observer based on the leader state information is designed.Then,a finite-time prescribed performance adaptive output feedback control strategy is proposed by restricting the sliding mode surface to a prescribed region,which ensures that the closed-loop system has practical finite-time stability and that formation errors of the multi-agent systems converge to the prescribed performance bound in finite time.Finally,a numerical simulation is provided to demonstrate the practicality and effectiveness of the developed algorithm.

Multi-scale physics-informed neural networks for solving high Reynolds number boundary layer flows based on matched asymptotic expansions

Multi-scale system remains a classical scientific problem in fluid dynamics,biology,etc.In the present study,a scheme of multi-scale Physics-informed neural networks is proposed to solve the boundary layer flow at high Reynolds numbers without any data.The flow is divided into several regions with different scales based on Prandtl's boundary theory.Different regions are solved with governing equations in different scales.The method of matched asymptotic expansions is used to make the flow field continuously.A flow on a semi infinite flat plate at a high Reynolds number is considered a multi-scale problem because the boundary layer scale is much smaller than the outer flow scale.The results are compared with the reference numerical solutions,which show that the msPINNs can solve the multi-scale problem of the boundary layer in high Reynolds number flows.This scheme can be developed for more multi-scale problems in the future.

Non-crossing Quantile Regression Neural Network as a Calibration Tool for Ensemble Weather Forecasts

Despite the maturity of ensemble numerical weather prediction(NWP),the resulting forecasts are still,more often than not,under-dispersed.As such,forecast calibration tools have become popular.Among those tools,quantile regression(QR)is highly competitive in terms of both flexibility and predictive performance.Nevertheless,a long-standing problem of QR is quantile crossing,which greatly limits the interpretability of QR-calibrated forecasts.On this point,this study proposes a non-crossing quantile regression neural network(NCQRNN),for calibrating ensemble NWP forecasts into a set of reliable quantile forecasts without crossing.The overarching design principle of NCQRNN is to add on top of the conventional QRNN structure another hidden layer,which imposes a non-decreasing mapping between the combined output from nodes of the last hidden layer to the nodes of the output layer,through a triangular weight matrix with positive entries.The empirical part of the work considers a solar irradiance case study,in which four years of ensemble irradiance forecasts at seven locations,issued by the European Centre for Medium-Range Weather Forecasts,are calibrated via NCQRNN,as well as via an eclectic mix of benchmarking models,ranging from the naïve climatology to the state-of-the-art deep-learning and other non-crossing models.Formal and stringent forecast verification suggests that the forecasts post-processed via NCQRNN attain the maximum sharpness subject to calibration,amongst all competitors.Furthermore,the proposed conception to resolve quantile crossing is remarkably simple yet general,and thus has broad applicability as it can be integrated with many shallow-and deep-learning-based neural networks.

Numerical Study of the Biomechanical Behavior of a 3D Printed Polymer Esophageal Stent in the Esophagus by BP Neural Network Algorithm

Esophageal disease is a common disorder of the digestive system that can severely affect the quality of life andprognosis of patients. Esophageal stenting is an effective treatment that has been widely used in clinical practice.However, esophageal stents of different types and parameters have varying adaptability and effectiveness forpatients, and they need to be individually selected according to the patient’s specific situation. The purposeof this study was to provide a reference for clinical doctors to choose suitable esophageal stents. We used 3Dprinting technology to fabricate esophageal stents with different ratios of thermoplastic polyurethane (TPU)/(Poly-ε-caprolactone) PCL polymer, and established an artificial neural network model that could predict the radial forceof esophageal stents based on the content of TPU, PCL and print parameter. We selected three optimal ratios formechanical performance tests and evaluated the biomechanical effects of different ratios of stents on esophagealimplantation, swallowing, and stent migration processes through finite element numerical simulation and in vitrosimulation tests. The results showed that different ratios of polymer stents had different mechanical properties,affecting the effectiveness of stent expansion treatment and the possibility of postoperative complications of stentimplantation.

Development of a convolutional neural network based geomechanical upscaling technique for heterogeneous geological reservoir

Geomechanical assessment using coupled reservoir-geomechanical simulation is becoming increasingly important for analyzing the potential geomechanical risks in subsurface geological developments.However,a robust and efficient geomechanical upscaling technique for heterogeneous geological reservoirs is lacking to advance the applications of three-dimensional(3D)reservoir-scale geomechanical simulation considering detailed geological heterogeneities.Here,we develop convolutional neural network(CNN)proxies that reproduce the anisotropic nonlinear geomechanical response caused by lithological heterogeneity,and compute upscaled geomechanical properties from CNN proxies.The CNN proxies are trained using a large dataset of randomly generated spatially correlated sand-shale realizations as inputs and simulation results of their macroscopic geomechanical response as outputs.The trained CNN models can provide the upscaled shear strength(R^(2)>0.949),stress-strain behavior(R^(2)>0.925),and volumetric strain changes(R^(2)>0.958)that highly agree with the numerical simulation results while saving over two orders of magnitude of computational time.This is a major advantage in computing the upscaled geomechanical properties directly from geological realizations without the need to perform local numerical simulations to obtain the geomechanical response.The proposed CNN proxybased upscaling technique has the ability to(1)bridge the gap between the fine-scale geocellular models considering geological uncertainties and computationally efficient geomechanical models used to assess the geomechanical risks of large-scale subsurface development,and(2)improve the efficiency of numerical upscaling techniques that rely on local numerical simulations,leading to significantly increased computational time for uncertainty quantification using numerous geological realizations.

Customized Convolutional Neural Network for Accurate Detection of Deep Fake Images in Video Collections

The motivation for this study is that the quality of deep fakes is constantly improving,which leads to the need to develop new methods for their detection.The proposed Customized Convolutional Neural Network method involves extracting structured data from video frames using facial landmark detection,which is then used as input to the CNN.The customized Convolutional Neural Network method is the date augmented-based CNN model to generate‘fake data’or‘fake images’.This study was carried out using Python and its libraries.We used 242 films from the dataset gathered by the Deep Fake Detection Challenge,of which 199 were made up and the remaining 53 were real.Ten seconds were allotted for each video.There were 318 videos used in all,199 of which were fake and 119 of which were real.Our proposedmethod achieved a testing accuracy of 91.47%,loss of 0.342,and AUC score of 0.92,outperforming two alternative approaches,CNN and MLP-CNN.Furthermore,our method succeeded in greater accuracy than contemporary models such as XceptionNet,Meso-4,EfficientNet-BO,MesoInception-4,VGG-16,and DST-Net.The novelty of this investigation is the development of a new Convolutional Neural Network(CNN)learning model that can accurately detect deep fake face photos.

Research on a TOPSIS energy efficiency evaluation system for crude oil gathering and transportation systems based on a GA-BP neural network

As the main link of ground engineering,crude oil gathering and transportation systems require huge energy consumption and complex structures.It is necessary to establish an energy efficiency evaluation system for crude oil gathering and transportation systems and identify the energy efficiency gaps.In this paper,the energy efficiency evaluation system of the crude oil gathering and transportation system in an oilfield in western China is established.Combined with the big data analysis method,the GA-BP neural network is used to establish the energy efficiency index prediction model for crude oil gathering and transportation systems.The comprehensive energy consumption,gas consumption,power consumption,energy utilization rate,heat utilization rate,and power utilization rate of crude oil gathering and transportation systems are predicted.Considering the efficiency and unit consumption index of the crude oil gathering and transportation system,the energy efficiency evaluation system of the crude oil gathering and transportation system is established based on a game theory combined weighting method and TOPSIS evaluation method,and the subjective weight is determined by the triangular fuzzy analytic hierarchy process.The entropy weight method determines the objective weight,and the combined weight of game theory combines subjectivity with objectivity to comprehensively evaluate the comprehensive energy efficiency of crude oil gathering and transportation systems and their subsystems.Finally,the weak links in energy utilization are identified,and energy conservation and consumption reduction are improved.The above research provides technical support for the green,efficient and intelligent development of crude oil gathering and transportation systems.

Trajectory tracking guidance of interceptor via prescribed performance integral sliding mode with neural network disturbance observer

This paper investigates interception missiles’trajectory tracking guidance problem under wind field and external disturbances in the boost phase.Indeed,the velocity control in such trajectory tracking guidance systems of missiles is challenging.As our contribution,the velocity control channel is designed to deal with the intractable velocity problem and improve tracking accuracy.The global prescribed performance function,which guarantees the tracking error within the set range and the global convergence of the tracking guidance system,is first proposed based on the traditional PPF.Then,a tracking guidance strategy is derived using the integral sliding mode control techniques to make the sliding manifold and tracking errors converge to zero and avoid singularities.Meanwhile,an improved switching control law is introduced into the designed tracking guidance algorithm to deal with the chattering problem.A back propagation neural network(BPNN)extended state observer(BPNNESO)is employed in the inner loop to identify disturbances.The obtained results indicate that the proposed tracking guidance approach achieves the trajectory tracking guidance objective without and with disturbances and outperforms the existing tracking guidance schemes with the lowest tracking errors,convergence times,and overshoots.

Intrusion Detection System for Smart Industrial Environments with Ensemble Feature Selection and Deep Convolutional Neural Networks

Smart Industrial environments use the Industrial Internet of Things(IIoT)for their routine operations and transform their industrial operations with intelligent and driven approaches.However,IIoT devices are vulnerable to cyber threats and exploits due to their connectivity with the internet.Traditional signature-based IDS are effective in detecting known attacks,but they are unable to detect unknown emerging attacks.Therefore,there is the need for an IDS which can learn from data and detect new threats.Ensemble Machine Learning(ML)and individual Deep Learning(DL)based IDS have been developed,and these individual models achieved low accuracy;however,their performance can be improved with the ensemble stacking technique.In this paper,we have proposed a Deep Stacked Neural Network(DSNN)based IDS,which consists of two stacked Convolutional Neural Network(CNN)models as base learners and Extreme Gradient Boosting(XGB)as the meta learner.The proposed DSNN model was trained and evaluated with the next-generation dataset,TON_IoT.Several pre-processing techniques were applied to prepare a dataset for the model,including ensemble feature selection and the SMOTE technique.Accuracy,precision,recall,F1-score,and false positive rates were used to evaluate the performance of the proposed ensemble model.Our experimental results showed that the accuracy for binary classification is 99.61%,which is better than in the baseline individual DL and ML models.In addition,the model proposed for IDS has been compared with similar models.The proposed DSNN achieved better performance metrics than the other models.The proposed DSNN model will be used to develop enhanced IDS for threat mitigation in smart industrial environments.

An End-To-End Hyperbolic Deep Graph Convolutional Neural Network Framework

Graph Convolutional Neural Networks(GCNs)have been widely used in various fields due to their powerful capabilities in processing graph-structured data.However,GCNs encounter significant challenges when applied to scale-free graphs with power-law distributions,resulting in substantial distortions.Moreover,most of the existing GCN models are shallow structures,which restricts their ability to capture dependencies among distant nodes and more refined high-order node features in scale-free graphs with hierarchical structures.To more broadly and precisely apply GCNs to real-world graphs exhibiting scale-free or hierarchical structures and utilize multi-level aggregation of GCNs for capturing high-level information in local representations,we propose the Hyperbolic Deep Graph Convolutional Neural Network(HDGCNN),an end-to-end deep graph representation learning framework that can map scale-free graphs from Euclidean space to hyperbolic space.In HDGCNN,we define the fundamental operations of deep graph convolutional neural networks in hyperbolic space.Additionally,we introduce a hyperbolic feature transformation method based on identity mapping and a dense connection scheme based on a novel non-local message passing framework.In addition,we present a neighborhood aggregation method that combines initial structural featureswith hyperbolic attention coefficients.Through the above methods,HDGCNN effectively leverages both the structural features and node features of graph data,enabling enhanced exploration of non-local structural features and more refined node features in scale-free or hierarchical graphs.Experimental results demonstrate that HDGCNN achieves remarkable performance improvements over state-ofthe-art GCNs in node classification and link prediction tasks,even when utilizing low-dimensional embedding representations.Furthermore,when compared to shallow hyperbolic graph convolutional neural network models,HDGCNN exhibits notable advantages and performance enhancements.

A Review of Computing with Spiking Neural Networks

Artificial neural networks(ANNs)have led to landmark changes in many fields,but they still differ significantly fromthemechanisms of real biological neural networks and face problems such as high computing costs,excessive computing power,and so on.Spiking neural networks(SNNs)provide a new approach combined with brain-like science to improve the computational energy efficiency,computational architecture,and biological credibility of current deep learning applications.In the early stage of development,its poor performance hindered the application of SNNs in real-world scenarios.In recent years,SNNs have made great progress in computational performance and practicability compared with the earlier research results,and are continuously producing significant results.Although there are already many pieces of literature on SNNs,there is still a lack of comprehensive review on SNNs from the perspective of improving performance and practicality as well as incorporating the latest research results.Starting from this issue,this paper elaborates on SNNs along the complete usage process of SNNs including network construction,data processing,model training,development,and deployment,aiming to provide more comprehensive and practical guidance to promote the development of SNNs.Therefore,the connotation and development status of SNNcomputing is reviewed systematically and comprehensively from four aspects:composition structure,data set,learning algorithm,software/hardware development platform.Then the development characteristics of SNNs in intelligent computing are summarized,the current challenges of SNNs are discussed and the future development directions are also prospected.Our research shows that in the fields of machine learning and intelligent computing,SNNs have comparable network scale and performance to ANNs and the ability to challenge large datasets and a variety of tasks.The advantages of SNNs over ANNs in terms of energy efficiency and spatial-temporal data processing have been more fully exploited.And the development of programming and deployment tools has lowered the threshold for the use of SNNs.SNNs show a broad development prospect for brain-like computing.

NeurstrucEnergy:A bi-directional GNN model for energy prediction of neural networks in IoT

A significant demand rises for energy-efficient deep neural networks to support power-limited embedding devices with successful deep learning applications in IoT and edge computing fields.An accurate energy prediction approach is critical to provide measurement and lead optimization direction.However,the current energy prediction approaches lack accuracy and generalization ability due to the lack of research on the neural network structure and the excessive reliance on customized training dataset.This paper presents a novel energy prediction model,NeurstrucEnergy.NeurstrucEnergy treats neural networks as directed graphs and applies a bi-directional graph neural network training on a randomly generated dataset to extract structural features for energy prediction.NeurstrucEnergy has advantages over linear approaches because the bi-directional graph neural network collects structural features from each layer's parents and children.Experimental results show that NeurstrucEnergy establishes state-of-the-art results with mean absolute percentage error of 2.60%.We also evaluate NeurstrucEnergy in a randomly generated dataset,achieving the mean absolute percentage error of 4.83%over 10 typical convolutional neural networks in recent years and 7 efficient convolutional neural networks created by neural architecture search.Our code is available at https://github.com/NEUSoftGreenAI/NeurstrucEnergy.git.

Expression Recognition Method Based on Convolutional Neural Network and Capsule Neural Network

Convolutional neural networks struggle to accurately handle changes in angles and twists in the direction of images,which affects their ability to recognize patterns based on internal feature levels. In contrast, CapsNet overcomesthese limitations by vectorizing information through increased directionality and magnitude, ensuring that spatialinformation is not overlooked. Therefore, this study proposes a novel expression recognition technique calledCAPSULE-VGG, which combines the strengths of CapsNet and convolutional neural networks. By refining andintegrating features extracted by a convolutional neural network before introducing theminto CapsNet, ourmodelenhances facial recognition capabilities. Compared to traditional neural network models, our approach offersfaster training pace, improved convergence speed, and higher accuracy rates approaching stability. Experimentalresults demonstrate that our method achieves recognition rates of 74.14% for the FER2013 expression dataset and99.85% for the CK+ expression dataset. By contrasting these findings with those obtained using conventionalexpression recognition techniques and incorporating CapsNet’s advantages, we effectively address issues associatedwith convolutional neural networks while increasing expression identification accuracy.

Assessment of Seasonal Rainfall Prediction in Ethiopia: Evaluating a Dynamic Recurrent Neural Network to Downscale ECMWF-SEAS5 Rainfall

Seasonal rainfall plays a vital role in both environmental dynamics and decision-making for rainfed agriculture in Ethiopia, a country often impacted by extreme climate events such as drought and flooding. Predicting the onset of the rainy season and providing localized rainfall forecasts for Ethiopia is challenging due to the changing spatiotemporal patterns and the country's rugged topography. The Climate Hazards Group Infra Red Precipitation with Station Data(CHIRPS), ERA5-Land total precipitation and temperature data are used from 1981–2022 to predict spatial rainfall by applying an artificial neural network(ANN). The recurrent neural network(RNN) is a nonlinear autoregressive network with exogenous input(NARX), which includes feed-forward connections and multiple network layers, employing the Levenberg Marquart algorithm. This method is applied to downscale data from the European Centre for Medium-range Weather Forecasts fifth-generation seasonal forecast system(ECMWF-SEAS5) and the Euro-Mediterranean Centre for Climate Change(CMCC) to the specific locations of rainfall stations in Ethiopia for the period 1980–2020. Across the stations, the results of NARX exhibit strong associations and reduced errors. The statistical results indicate that, except for the southwestern Ethiopian highlands, the downscaled monthly precipitation data exhibits high skill scores compared to the station records, demonstrating the effectiveness of the NARX approach for predicting local seasonal rainfall in Ethiopia's complex terrain. In addition to this spatial ANN of the summer season precipitation, temperature, as well as the combination of these two variables, show promising results.