Optimized operation scheme of flash-memory-based neural network online training with ultra-high endurance

With the rapid development of machine learning,the demand for high-efficient computing becomes more and more urgent.To break the bottleneck of the traditional Von Neumann architecture,computing-in-memory(CIM)has attracted increasing attention in recent years.In this work,to provide a feasible CIM solution for the large-scale neural networks(NN)requiring continuous weight updating in online training,a flash-based computing-in-memory with high endurance(10^(9) cycles)and ultrafast programming speed is investigated.On the one hand,the proposed programming scheme of channel hot electron injection(CHEI)and hot hole injection(HHI)demonstrate high linearity,symmetric potentiation,and a depression process,which help to improve the training speed and accuracy.On the other hand,the low-damage programming scheme and memory window(MW)optimizations can suppress cell degradation effectively with improved computing accuracy.Even after 109 cycles,the leakage current(I_(off))of cells remains sub-10pA,ensuring the large-scale computing ability of memory.Further characterizations are done on read disturb to demonstrate its robust reliabilities.By processing CIFAR-10 tasks,it is evident that~90%accuracy can be achieved after 109 cycles in both ResNet50 and VGG16 NN.Our results suggest that flash-based CIM has great potential to overcome the limitations of traditional Von Neumann architectures and enable high-performance NN online training,which pave the way for further development of artificial intelligence(AI)accelerators.

NeTrainSim:a network-level simulator for modeling freight train longitudinal motion and energy consumption

Although train modeling research is vast, most available simulation tools are confined to city-or trip-scale analysis, primarily offering micro-level simulations of network segments. This paper addresses this void by developing the Ne Train Sim simulator for heavy long-haul freight trains on a network of multiple intersecting tracks. The main objective of this simulator is to enable a comprehensive analysis of energy consumption and the associated carbon footprint for the entire train system. Four case studies were conducted to demonstrate the simulator's performance. The first case study validates the model by comparing Ne Train Sim output to empirical trajectory data. The results demonstrate that the simulated trajectory is precise enough to estimate the train energy consumption and carbon dioxide emissions. The second application demonstrates the train-following model considering six trains following each other. The results showcase the model ability to maintain safefollowing distances between successive trains. The next study highlights the simulator's ability to resolve train conflicts for different scenarios. Finally, the suitability of the Ne Train Sim for modeling realistic railroad networks is verified through the modeling of the entire US network and comparing alternative powertrains on the fleet energy consumption.

Effect of cognitive training on brain dynamics

The human brain is highly plastic.Cognitive training is usually used to modify functional connectivity of brain networks.Moreover,the structures of brain networks may determine its dynamic behavior which is related to human cognitive abilities.To study the effect of functional connectivity on the brain dynamics,the dynamic model based on functional connections of the brain and the Hindmarsh–Rose model is utilized in this work.The resting-state fMRI data from the experimental group undergoing abacus-based mental calculation(AMC)training and from the control group are used to construct the functional brain networks.The dynamic behavior of brain at the resting and task states for the AMC group and the control group are simulated with the above-mentioned dynamic model.In the resting state,there are the differences of brain activation between the AMC group and the control group,and more brain regions are inspired in the AMC group.A stimulus with sinusoidal signals to brain networks is introduced to simulate the brain dynamics in the task states.The dynamic characteristics are extracted by the excitation rates,the response intensities and the state distributions.The change in the functional connectivity of brain networks with the AMC training would in turn improve the brain response to external stimulus,and make the brain more efficient in processing tasks.

Respiratory training interventions improve health status of heart failure patients: A systematic review and network meta-analysis of randomized controlled trials

BACKGROUND Prior studies indicate that doing breathing exercises improves physical performance and quality of life (QoL) in heart failure patients. However, these effects remain unclear and contradictory. AIM To determine the effects of machine-assisted and non-machine-assisted respiratory training on physical performance and QoL in heart failure patients. METHODS This was a systematic review and network meta-analysis study. A literature search of electronic databases was conducted for randomized controlled trials (RCTs) on heart failure. Respiratory training interventions were grouped as seven categories: IMT_Pn (inspiratory muscle training without pressure or < 10% maximal inspiratory pressure, MIP), IMT_Pl (inspiratory muscle training with low pressure, 10%-15% MIP), IMT_Pm (inspiratory muscle training with medium pressure, 30%-40% MIP), IMT_Ph (inspiratory muscle training with high pressure, 60% MIP or MIP plus aerobics), Aerobics (aerobic exercise or weight training), Qi_Ex (tai chi, yoga, and breathing exercise), and none. The four outcomes were heart rate, peak oxygen uptake (VO2 peak), 6-min walking distance test (6MWT), and Minnesota Living with Heart Failure QoL. The random-effects model, side-splitting model, and the surface under the cumulative ranking curve (SUCRA) were used to test and analyze the data. RESULTS A total of 1499 subjects from 31 RCT studies were included. IMT_Ph had the highest effect sizes for VO2 peak and 6MWT, IMT_Pm highest for QoL, and Qi_Ex highest for heart rate. Aerobics had the second highest for VO2 peak, Qi_Ex second highest for 6MWT, and IMT_Ph second highest for heart rate and QoL.CONCLUSION This study supports that high- and medium-intensity machine-assisted training improves exercise capacity and QoL in hospital-based heart failure patients. After hospital discharge, non-machine-assisted training continuously improves cardiac function.

Towards efficient deep neural network training by FPGA-based batch-level parallelism

Training deep neural networks(DNNs)requires a significant amount of time and resources to obtain acceptable results,which severely limits its deployment in resource-limited platforms.This paper proposes DarkFPGA,a novel customizable framework to efficiently accelerate the entire DNN training on a single FPGA platform.First,we explore batch-level parallelism to enable efficient FPGA-based DNN training.Second,we devise a novel hardware architecture optimised by a batch-oriented data pattern and tiling techniques to effectively exploit parallelism.Moreover,an analytical model is developed to determine the optimal design parameters for the DarkFPGA accelerator with respect to a specific network specification and FPGA resource constraints.Our results show that the accelerator is able to perform about 10 times faster than CPU training and about a third of the energy consumption than GPU training using 8-bit integers for training VGG-like networks on the CIFAR dataset for the Maxeler MAX5 platform.

Dimensionality Reduction with Input Training Neural Network and Its Application in Chemical Process Modelling

Many applications of principal component analysis (PCA) can be found in dimensionality reduction. But linear PCA method is not well suitable for nonlinear chemical processes. A new PCA method based on im-proved input training neural network (IT-NN) is proposed for the nonlinear system modelling in this paper. Mo-mentum factor and adaptive learning rate are introduced into learning algorithm to improve the training speed of IT-NN. Contrasting to the auto-associative neural network (ANN), IT-NN has less hidden layers and higher training speed. The effectiveness is illustrated through a comparison of IT-NN with linear PCA and ANN with experiments. Moreover, the IT-NN is combined with RBF neural network (RBF-NN) to model the yields of ethylene and propyl-ene in the naphtha pyrolysis system. From the illustrative example and practical application, IT-NN combined with RBF-NN is an effective method of nonlinear chemical process modelling.

Sampling Methods for Efficient Training of Graph Convolutional Networks:A Survey

Graph convolutional networks(GCNs)have received significant attention from various research fields due to the excellent performance in learning graph representations.Although GCN performs well compared with other methods,it still faces challenges.Training a GCN model for large-scale graphs in a conventional way requires high computation and storage costs.Therefore,motivated by an urgent need in terms of efficiency and scalability in training GCN,sampling methods have been proposed and achieved a significant effect.In this paper,we categorize sampling methods based on the sampling mechanisms and provide a comprehensive survey of sampling methods for efficient training of GCN.To highlight the characteristics and differences of sampling methods,we present a detailed comparison within each category and further give an overall comparative analysis for the sampling methods in all categories.Finally,we discuss some challenges and future research directions of the sampling methods.

Judicious training pattern for superior molecular reorganization energy prediction model

Reorganization energy(RE)is closely related to the charge transport properties and is one of the important parameters for screening novel organic semiconductors(OSCs).With the rise of data-driven technology,accurate and efficient machine learning(ML)models for high-throughput screening novel organic molecules play an important role in the boom of material science.Comparing different molecular descriptors and algorithms,we construct a reasonable algorithm framework with molecular graphs to describe the compositional structure,convolutional neural networks to extract material features,and subsequently embedded fully connected neural networks to establish the mapping between features and predicted properties.With our well-designed judicious training pattern about feature-guided stratified random sampling,we have obtained a high-precision and robust reorganization energy prediction model,which can be used as one of the important descriptors for rapid screening potential OSCs.The root-meansquare error(RMSE)and the squared Pearson correlation coefficient(R^(2))of this model are 2.6 me V and0.99,respectively.More importantly,we confirm and emphasize that training pattern plays a crucial role in constructing supreme ML models.We are calling for more attention to designing innovative judicious training patterns in addition to high-quality databases,efficient material feature engineering and algorithm framework construction.

Effect of endurance training on the expression profile of circRNA-lncRNA-miRNA-mRNA in myocardial tissues of mice after exhaustive exercise

Objective:To clarify the effect of endurance training on the expression profile of circRNA-lncRNA-miRNA-mRNA in myocardial tissues of mice after exhaustive exercise.Methods:A total of 45 male C57BL/6 mice were randomly divided into control(C),low-strength endurance training(LSET)and high-strength endurance training(HSET)groups(n=15).The mice in the control group were not conducted to platform training.The mice in the LSET and HSET groups were conducted to platform training at 30%and 60%of exhaustive exercise once a day for 5 days a week,respectively.The exhaustion exercise was performed after 5 weeks of platform training.Total RNA was extracted from myocardial tissues,and the expression profile of circRNA-lncRNA-miRNA-mRNA in myocardial tissues was analyzed using Illimina transcriptome sequencing.Results:The distance and time of exhaustive exercise were longer in the LSET and HSET groups than in the control group,and the distance and time of exhaustive exercise were longer in the HSET group than in the LSET group(P<0.05).A total of 54 differentially expressed circRNAs(28 down-regulated and 26 up-regulated),7 differentially expressed lncRNAs(all down-regulated),3 differentially expressed miRNAs(1 down-regulated and 2 up-regulated)and 99 differentially expressed mRNAs(81 down-regulated and 18 up-regulated)were identified by transcriptome sequencing(P<0.05).Interaction network analysis revealed that ENSMUSG00000113041,MSTRG.79740,mmu-miR-374c-5p,18 down-regulated mRNAs and 3 up-regulated mRNAs formed a regulatory network.GO functional analysis revealed that the differentially expressed mRNAs were mainly enriched in primary metabolic processes and macromolecular synthesis and metabolic processes.KEGG pathway analysis revealed that the differentially expressed mRNAs were mainly enriched in complement and coagulation cascade pathways,estrogen signaling pathway and glucagon signaling pathway.Conclusion:Endurance training could alter the expression profile of circRNA-lncRNA-miRNA-mRNA in myocardial tissues of mice after exhaustive exercise,and these differentially expressed RNAs form a regulatory network that affects cardiomyocyte synthesis and metabolism and thus participates in the regulation of myocardial injury.

Efficient Training of Multi-Layer Neural Networks to Achieve Faster Validation

Artificial neural networks(ANNs)are one of the hottest topics in computer science and artificial intelligence due to their potential and advantages in analyzing real-world problems in various disciplines,including but not limited to physics,biology,chemistry,and engineering.However,ANNs lack several key characteristics of biological neural networks,such as sparsity,scale-freeness,and small-worldness.The concept of sparse and scale-free neural networks has been introduced to fill this gap.Network sparsity is implemented by removing weak weights between neurons during the learning process and replacing them with random weights.When the network is initialized,the neural network is fully connected,which means the number of weights is four times the number of neurons.In this study,considering that a biological neural network has some degree of initial sparsity,we design an ANN with a prescribed level of initial sparsity.The neural network is tested on handwritten digits,Arabic characters,CIFAR-10,and Reuters newswire topics.Simulations show that it is possible to reduce the number of weights by up to 50%without losing prediction accuracy.Moreover,in both cases,the testing time is dramatically reduced compared with fully connected ANNs.

Altered electroencephalographic networks in developmental dyslexia after remedial training:a prospective case-control study

Electroencephalographic studies using graph theoretic analysis have found aberrations in functional connectivity in children with developmental dyslexia.However,how the training with visual tasks can change the functional connectivity of the semantic network in developmental dyslexia is still unclear.We looked for differences in local and global topological properties of functional networks between 21 healthy controls and 22 dyslexic children(8–9 years old)before and after training with visual tasks in this prospective case-control study.The minimum spanning tree method was used to construct the subjects’brain networks in multiple electroencephalographic frequency ranges during a visual word/pseudoword discrimination task.We found group differences in the theta,alpha,beta and gamma bands for four graph measures suggesting a more integrated network topology in dyslexics before the training compared to controls.After training,the network topology of dyslexic children had become more segregated and similar to that of the controls.In theθ,αandβ1-frequency bands,compared to the controls,the pre-training dyslexics exhibited a reduced degree and betweenness centrality of the left anterior temporal and parietal regions.The simultaneous appearance in the left hemisphere of hubs in temporal and parietal(α,β1),temporal and superior frontal cortex(θ,α),parietal and occipitotemporal cortices(β1),identified in the networks of normally developing children was not present in the brain networks of dyslexics.After training,the hub distribution for dyslexics in the theta and beta1 bands had become similar to that of the controls.In summary,our findings point to a less efficient network configuration in dyslexics compared to a more optimal global organization in the controls.This is the first study to investigate the topological organization of functional brain networks of Bulgarian dyslexic children.Approval for the study was obtained from the Ethics Committee of the Institute of Neurobiology and the Institute for Population and Human Studies,Bulgarian Academy of Sciences(approval No.02-41/12.07.2019)on March 28,2017,and the State Logopedic Center and the Ministry of Education and Science(approval No.09-69/14.03.2017)on July 12,2019.

Application of Convolutional Neural Networks in Classification of GBM for Enhanced Prognosis

The lethal brain tumor “Glioblastoma” has the propensity to grow over time. To improve patient outcomes, it is essential to classify GBM accurately and promptly in order to provide a focused and individualized treatment plan. Despite this, deep learning methods, particularly Convolutional Neural Networks (CNNs), have demonstrated a high level of accuracy in a myriad of medical image analysis applications as a result of recent technical breakthroughs. The overall aim of the research is to investigate how CNNs can be used to classify GBMs using data from medical imaging, to improve prognosis precision and effectiveness. This research study will demonstrate a suggested methodology that makes use of the CNN architecture and is trained using a database of MRI pictures with this tumor. The constructed model will be assessed based on its overall performance. Extensive experiments and comparisons with conventional machine learning techniques and existing classification methods will also be made. It will be crucial to emphasize the possibility of early and accurate prediction in a clinical workflow because it can have a big impact on treatment planning and patient outcomes. The paramount objective is to not only address the classification challenge but also to outline a clear pathway towards enhancing prognosis precision and treatment effectiveness.

Convergence of Hyperbolic Neural Networks Under Riemannian Stochastic Gradient Descent

We prove,under mild conditions,the convergence of a Riemannian gradient descent method for a hyperbolic neural network regression model,both in batch gradient descent and stochastic gradient descent.We also discuss a Riemannian version of the Adam algorithm.We show numerical simulations of these algorithms on various benchmarks.

A Second Order Training Algorithm for Multilayer Feedforward Neural Networks

ＡＳｅｃｏｎｄＯｒｄｅｒＴｒａｉｎｉｎｇＡｌｇｏｒｉｔｈｍｆｏｒＭｕｌｔｉｌａｙｅｒＦｅｅｄｆｏｒｗａｒｄＮｅｕｒａｌＮｅｔｗｏｒｋｓＴａｎＹｉｎｇ（谭营）ＨｅＺｈｅｎｙａ（何振亚）（ＤｅｐａｒｔｍｅｎｔｏｆＲａｄｉｏＥｎｇｉｎｅｅｒｉｎｇ，Ｓｏｕ...

Time-varying parameters estimation with adaptive neural network EKF for missile-dual control system

In this paper, a filtering method is presented to estimate time-varying parameters of a missile dual control system with tail fins and reaction jets as control variables. In this method, the long-short-term memory(LSTM) neural network is nested into the extended Kalman filter(EKF) to modify the Kalman gain such that the filtering performance is improved in the presence of large model uncertainties. To avoid the unstable network output caused by the abrupt changes of system states,an adaptive correction factor is introduced to correct the network output online. In the process of training the network, a multi-gradient descent learning mode is proposed to better fit the internal state of the system, and a rolling training is used to implement an online prediction logic. Based on the Lyapunov second method, we discuss the stability of the system, the result shows that when the training error of neural network is sufficiently small, the system is asymptotically stable. With its application to the estimation of time-varying parameters of a missile dual control system, the LSTM-EKF shows better filtering performance than the EKF and adaptive EKF(AEKF) when there exist large uncertainties in the system model.

Towards high performance low bitwidth training for deep neural networks

The high performance of the state-of-the-art deep neural networks(DNNs)is acquired at the cost of huge consumption of computing resources.Quantization of networks is recently recognized as a promising solution to solve the problem and significantly reduce the resource usage.However,the previous quantization works have mostly focused on the DNN inference,and there were very few works to address on the challenges of DNN training.In this paper,we leverage dynamic fixed-point(DFP)quantization algorithm and stochastic rounding(SR)strategy to develop a fully quantized 8-bit neural networks targeting low bitwidth training.The experiments show that,in comparison to the full-precision networks,the accuracy drop of our quantized convolutional neural networks(CNNs)can be less than 2%,even when applied to deep models evaluated on Image-Net dataset.Additionally,our 8-bit GNMT translation network can achieve almost identical BLEU to full-precision network.We further implement a prototype on FPGA and the synthesis shows that the low bitwidth training scheme can reduce the resource usage significantly.

Short-term train arrival delay prediction:a data-driven approach

Purpose-To optimize train operations,dispatchers currently rely on experience for quick adjustments when delays occur.However,delay predictions often involve imprecise shifts based on known delay times.Real-time and accurate train delay predictions,facilitated by data-driven neural network models,can significantly reduce dispatcher stress and improve adjustment plans.Leveraging current train operation data,these models enable swift and precise predictions,addressing challenges posed by train delays in high-speed rail networks during unforeseen events.Design/methodology/approach-This paper proposes CBLA-net,a neural network architecture for predicting late arrival times.It combines CNN,Bi-LSTM,and attention mechanisms to extract features,handle time series data,and enhance information utilization.Trained on operational data from the Beijing-Tianjin line,it predicts the late arrival time of a target train at the next station using multidimensional input data from the target and preceding trains.Findings-This study evaluates our model’s predictive performance using two data approaches:one considering full data and another focusing only on late arrivals.Results show precise and rapid predictions.Training with full data achieves aMAEof approximately 0.54 minutes and a RMSEof 0.65 minutes,surpassing the model trained solely on delay data(MAE:is about 1.02 min,RMSE:is about 1.52 min).Despite superior overall performance with full data,the model excels at predicting delays exceeding 15 minutes when trained exclusively on late arrivals.For enhanced adaptability to real-world train operations,training with full data is recommended.Originality/value-This paper introduces a novel neural network model,CBLA-net,for predicting train delay times.It innovatively compares and analyzes the model’s performance using both full data and delay data formats.Additionally,the evaluation of the network’s predictive capabilities considers different scenarios,providing a comprehensive demonstration of the model’s predictive performance.

A network-based cognitive training induces cognitive improvements and neuroplastic changes in patients with relapsing-remitting multiple sclerosis:an exploratory case-control study

Cognitive impairments are commonly observed in patients with multiple sclerosis and are associated with lower levels of quality of life.No consensus has been reached on how to tackle effectively cognitive decline in this clinical population non-pharmacologically.This exploratory case-control study aims to investigate the effectiveness of a hypothesis-based cognitive training designed to target multiple domains by promoting the synchronous co-activation of different brain areas and thereby improve cognition and induce changes in functional connectivity in patients with relapsing-remitting multiple sclerosis.Forty-five patients(36 females and 9 males,mean age 44.62±8.80 years)with clinically stable relapsing-remitting multiple sclerosis were assigned to either a standard cognitive training or to control groups(sham training and nonactive control).The standard training included twenty sessions of computerized exercises involving various cognitive functions supported by distinct brain networks.The sham training was a modified version of the standard training that comprised the same exercises and number of sessions but with increased processing speed load.The non-active control group received no cognitive training.All patients underwent comprehensive neuropsychological and magnetic resonance imaging assessments at baseline and after 5 weeks.Cognitive and resting-state magnetic resonance imaging data were analyzed using repeated measures models.At reassessment,the standard training group showed significant cognitive improvements compared to both control groups in memory tasks not specifically targeted by the training:the Buschke Selective Reminding Test and the Semantic Fluency test.The standard training group showed reductions in functional connectivity of the salience network,in the anterior cingulate cortex,associated with improvements on the Buschke Selective Reminding Test.No changes were observed in the sham training group.These findings suggest that multi-domain training that stimulates multiple brain areas synchronously may improve cognition in people with relapsing-remitting multiple sclerosis if sufficient time to process training material is allowed.The associated reduction in functional connectivity of the salience network suggests that training-induced neuroplastic functional reorganization may be the mechanism supporting performance gains.This study was approved by the Regional Ethics Committee of Yorkshire and Humber(approval No.12/YH/0474)on November 20,2013.

Long Short-Term Memory Recurrent Neural Network-Based Acoustic Model Using Connectionist Temporal Classification on a Large-Scale Training Corpus

A Long Short-Term Memory(LSTM) Recurrent Neural Network(RNN) has driven tremendous improvements on an acoustic model based on Gaussian Mixture Model(GMM). However, these models based on a hybrid method require a forced aligned Hidden Markov Model(HMM) state sequence obtained from the GMM-based acoustic model. Therefore, it requires a long computation time for training both the GMM-based acoustic model and a deep learning-based acoustic model. In order to solve this problem, an acoustic model using CTC algorithm is proposed. CTC algorithm does not require the GMM-based acoustic model because it does not use the forced aligned HMM state sequence. However, previous works on a LSTM RNN-based acoustic model using CTC used a small-scale training corpus. In this paper, the LSTM RNN-based acoustic model using CTC is trained on a large-scale training corpus and its performance is evaluated. The implemented acoustic model has a performance of 6.18% and 15.01% in terms of Word Error Rate(WER) for clean speech and noisy speech, respectively. This is similar to a performance of the acoustic model based on the hybrid method.

Effective distributed convolutional neural network architecture for remote sensing images target classification with a pre-training approach

How to recognize targets with similar appearances from remote sensing images(RSIs) effectively and efficiently has become a big challenge. Recently, convolutional neural network(CNN) is preferred in the target classification due to the powerful feature representation ability and better performance. However,the training and testing of CNN mainly rely on single machine.Single machine has its natural limitation and bottleneck in processing RSIs due to limited hardware resources and huge time consuming. Besides, overfitting is a challenge for the CNN model due to the unbalance between RSIs data and the model structure.When a model is complex or the training data is relatively small,overfitting occurs and leads to a poor predictive performance. To address these problems, a distributed CNN architecture for RSIs target classification is proposed, which dramatically increases the training speed of CNN and system scalability. It improves the storage ability and processing efficiency of RSIs. Furthermore,Bayesian regularization approach is utilized in order to initialize the weights of the CNN extractor, which increases the robustness and flexibility of the CNN model. It helps prevent the overfitting and avoid the local optima caused by limited RSI training images or the inappropriate CNN structure. In addition, considering the efficiency of the Na¨?ve Bayes classifier, a distributed Na¨?ve Bayes classifier is designed to reduce the training cost. Compared with other algorithms, the proposed system and method perform the best and increase the recognition accuracy. The results show that the distributed system framework and the proposed algorithms are suitable for RSIs target classification tasks.