FPGA Implementation of a Scalable and Highly Parallel Architecture for Restricted Boltzmann Machines

Restricted Boltzmann Machines (RBMs) are an effective model for machine learning;however, they require a significant amount of processing time. In this study, we propose a highly parallel, highly flexible architecture that combines small and completely parallel RBMs. This proposal addresses problems associated with calculation speed and exponential increases in circuit scale. We show that this architecture can optionally respond to the trade-offs between these two problems. Furthermore, our FPGA implementation performs at a 134 times processing speed up factor with respect to a conventional CPU.

Generalization properties of restricted Boltzmann machine for short-range order

A biased sampling algorithm for the restricted Boltzmann machine(RBM) is proposed, which allows generating configurations with a conserved quantity. To validate the method, a study of the short-range order in binary alloys with positive and negative exchange interactions is carried out. The network is trained on the data collected by Monte–Carlo simulations for a simple Ising-like binary alloy model and used to calculate the Warren–Cowley short-range order parameter and other thermodynamic properties. We demonstrate that the proposed method allows us not only to correctly reproduce the order parameters for the alloy concentration at which the network was trained, but can also predict them for any other concentrations.

Classification of steel based on laser-induced breakdown spectroscopy combined with restricted Boltzmann machine and support vector machine

In recent years,a laser-induced breakdown spectrometer(LIBS)combined with machine learning has been widely developed for steel classification.However,the much redundant information of LIBS spectra increases the computation complexity for classification.In this work,restricted Boltzmann machines(RBM)and principal component analysis(PCA)were used for dimension reduction of datasets,respectively.Then,a support vector machine(SVM)was adopted to process feature information.Two models(RBM-SVM and PCA-SVM)are compared in terms of performance.After optimization,the accuracy of the RBM-SVM model can achieve 100%,and the maximum dimension reduction time is 33.18 s,which is nearly half of that of the PCA model(53.19 s).These results preliminarily indicate that LIBS combined with RBM-SVM has great potential in the real-time classification of steel.

Sustainable Investment Forecasting of Power Grids Based on theDeep Restricted Boltzmann Machine Optimized by the Lion Algorithm

This paper proposes a new power grid investment prediction model based on the deep restricted Boltzmann machine(DRBM)optimized by the Lion algorithm(LA).Firstly,two factors including transmission and distribution price reform(TDPR)and 5G station construction were comprehensively incorporated into the consideration of influencing factors,and the fuzzy threshold method was used to screen out critical influencing factors.Then,the LA was used to optimize the parameters of the DRBM model to improve the model’s prediction accuracy,and the model was trained with the selected influencing factors and investment.Finally,the LA-DRBM model was used to predict the investment of a power grid enterprise,and the final prediction result was obtained by modifying the initial result with the modifying factors.The LA-DRBMmodel compensates for the deficiency of the singlemodel,and greatly improves the investment prediction accuracy of the power grid.In this study,a power grid enterprise was taken as an example to carry out an empirical analysis to prove the validity of the model,and a comparison with the RBM,support vector machine(SVM),back propagation neural network(BPNN),and regression model was conducted to verify the superiority of the model.The conclusion indicates that the proposed model has a strong generalization ability and good robustness,is able to abstract the combination of low-level features into high-level features,and can improve the efficiency of the model’s calculations for investment prediction of power grid enterprises.

Restricted Boltzmann machine: Recent advances and mean-field theory

This review deals with restricted Boltzmann machine(RBM) under the light of statistical physics.The RBM is a classical family of machine learning(ML) models which played a central role in the development of deep learning.Viewing it as a spin glass model and exhibiting various links with other models of statistical physics,we gather recent results dealing with mean-field theory in this context.First the functioning of the RBM can be analyzed via the phase diagrams obtained for various statistical ensembles of RBM,leading in particular to identify a compositional phase where a small number of features or modes are combined to form complex patterns.Then we discuss recent works either able to devise mean-field based learning algorithms;either able to reproduce generic aspects of the learning process from some ensemble dynamics equations or/and from linear stability arguments.

Green's function Monte Carlo method combined with restricted Boltzmann machine approach to the frustrated J_(1)–J_(2)Heisenberg model

Restricted Boltzmann machine(RBM)has been proposed as a powerful variational ansatz to represent the ground state of a given quantum many-body system.On the other hand,as a shallow neural network,it is found that the RBM is still hardly able to capture the characteristics of systems with large sizes or complicated interactions.In order to find a way out of the dilemma,here,we propose to adopt the Green's function Monte Carlo(GFMC)method for which the RBM is used as a guiding wave function.To demonstrate the implementation and effectiveness of the proposal,we have applied the proposal to study the frustrated J_(1)-J_(2)Heisenberg model on a square lattice,which is considered as a typical model with sign problem for quantum Monte Carlo simulations.The calculation results demonstrate that the GFMC method can significantly further reduce the relative error of the ground-state energy on the basis of the RBM variational results.This encourages to combine the GFMC method with other neural networks like convolutional neural networks for dealing with more models with sign problem in the future.

Monitoring and diagnosis of complex production process based on free energy of Gaussian–Bernoulli restricted Boltzmann machine

Online monitoring and diagnosis of production processes face great challenges due to the nonlinearity and multivariate of complex industrial processes.Traditional process monitoring methods employ kernel function or multilayer neural networks to solve the nonlinear mapping problem of data.However,the above methods increase the model complexity and are not interpretable,leading to difficulties in subsequent fault recognition/diagnosis/location.A process monitoring and diagnosis method based on the free energy of Gaussian-Bernoulli restricted Boltzmann machine(GBRBM-FE)was proposed.Firstly,a GBRBM network was established to make the probability distribution of the reconstructed data as close as possible to the probability distribution of the raw data.On this basis,the weights and biases in GBRBM network were used to construct F statistics,which represents the free energy of the sample.The smaller the energy of the sample is,the more normal the sample is.Therefore,F statistics can be used to monitor the production process.To diagnose fault variables,the F statistic for each sample was decomposed to obtain the Fv statistic for each variable.By analyzing the deviation degree between the corresponding variables of abnormal samples and normal samples,the cause of process abnormalities can be accurately located.The application of converter steelmaking process demonstrates that the proposed method outperforms the traditional methods,in terms of fault monitoring and diagnosis performance.

Intrusion detection model based on deep belief nets

This paper focuses on the intrusion classification of huge amounts of data in a network intrusion detection system. An intrusion detection model based on deep belief nets （DBN） is proposed to conduct intrusion detection,and the principles regarding DBN are discussed.The DBN is composed of a multiple unsupervised restricted Boltzmann machine （RBM） and a supervised back propagation （BP）network.First,the DBN in the proposed model is pre-trained in a fast and greedy way,and each RBM is trained by the contrastive divergence algorithm.Secondly,the whole network is fine-tuned by the supervised BP algorithm,which is employed for classifying the low-dimensional features of the intrusion data generated by the last RBM layer simultaneously.The experimental results on the KDD CUP 1999 dataset demonstrate that the DBN using the RBM network with three or more layers outperforms the self-organizing maps （SOM）and neural network （NN）in intrusion classification.Therefore,the DBN is an efficient approach for intrusion detection in high-dimensional space.

Boltzmann machines with clusters of stochastic binary units

The original restricted Boltzmann machines(RBMs)are extended by replacing the binary visible and hidden variables with clusters of binary units,and a new learning algorithm for training deep Boltzmann machine of this new variant is proposed.The sum of binary units of each cluster is approximated by a Gaussian distribution.Experiments demonstrate that the proposed Boltzmann machines can achieve good performance in the MNIST handwritten digital recognition task.

A new item-based deep network structure using a restricted Boltzmann machine for collaborative filtering

The collaborative filtering(CF) technique has been widely used recently in recommendation systems. It needs historical data to give predictions. However, the data sparsity problem still exists. We propose a new item-based restricted Boltzmann machine(RBM) approach for CF and use the deep multilayer RBM network structure, which alleviates the data sparsity problem and has excellent ability to extract features. Each item is treated as a single RBM, and different items share the same weights and biases. The parameters are learned layer by layer in the deep network. The batch gradient descent algorithm with minibatch is used to increase the convergence speed. The new feature vector discovered by the multilayer RBM network structure is very effective in predicting a rating and achieves a better result. Experimental results on the data set of MovieL ens show that the item-based multilayer RBM approach achieves the best performance, with a mean absolute error of 0.6424 and a root-mean-square error of 0.7843.

AN ENSEMBLE MODEL OF ARIMA AND ANN WITH RESTRICTED BOLTZMANN MACHINE BASED ON DECOMPOSITION OF DISCRETE WAVELET TRANSFORM FOR TIME SERIES FORECASTING

Time series forecasting research area mainly focuses on developing effective forecasting models toimprove prediction accuracy. An ensemble model composed of autoregressive integrated movingaverage （ARIMA）, artificial neural network （ANN）, restricted Boltzmann machines （RBM）, anddiscrete wavelet transform （DWT） is presented in this paper. In the proposed model, DWT firstdecomposes time series into approximation and detail. Then Khashei and Bijari＇s model, which is anensemble model of ARIMA and ANN, is applied to the approximation and detail to extract their bothlinear and nonlinear components and fit the relationship between the components as a function insteadof additive relationship. Furthermore, RBM is used to perform pre-training for generating initialweights and biases based on inputs feature for ANN. Finally, the forecasted approximation and detailare combined to obtain final forecasting. The forecasting capability of the proposed model is testedwith three well-known time series： sunspot, Canadian lynx, exchange rate time series. The predictionperformance is compared to the other six forecasting models. The results indicate that the proposedmodel gives the best performance in all three data sets and all three measures （i.e. MSE, MAE andMAPE）.

Damage identification of steel truss bridges based on deep belief network

To improve the accuracy and anti-noise ability of the structural damage identification method,a bridge damage identification method is proposed based on a deep belief network(DBN).The output vector is used to establish the nonlinear mapping relationship between the mode shape and structural damage.The hidden layer of the DBN is trained through a layer-by-layer pre-training.Finally,the backpropagation algorithm is used to fine-tune the entire network.The method is validated using a numerical model of a steel truss bridge.The results show that under the influence of noise and modeling uncertainty,the damage identification method based on the DBN can identify the accurate damage location and degree identification compared with the traditional damage identification method based on an artificial neural network.

Heart Disease Classification Using Multiple K-PCA and Hybrid Deep Learning Approach

One of the severe health problems and the most common types of heartdisease (HD) is Coronary heart disease (CHD). Due to the lack of a healthy lifestyle, HD would cause frequent mortality worldwide. If the heart attack occurswithout any symptoms, it cannot be cured by an intelligent detection system.An effective diagnosis and detection of CHD should prevent human casualties.Moreover, intelligent systems employ clinical-based decision support approachesto assist physicians in providing another option for diagnosing and detecting HD.This paper aims to introduce a heart disease prediction model including phaseslike (i) Feature extraction, (ii) Feature selection, and (iii) Classification. At first,the feature extraction process is carried out, where the features like a time-domainindex, frequency-domain index, geometrical domain features, nonlinear features,WT features, signal energy, skewness, entropy, kurtosis features are extractedfrom the input ECG signal. The curse of dimensionality becomes a severe issue.This paper provides the solution for this issue by introducing a new ModifiedPrincipal Component Analysis known as Multiple Kernel-based PCA for dimensionality reduction. Furthermore, the dimensionally reduced feature set is thensubjected to a classification process, where the hybrid classifier combining bothRecurrent Neural Network (RNN) and Restricted Boltzmann Machine (RBM)is used. At last, the performance analysis of the adopted scheme is compared overother existing schemes in terms of specific measures.

Cumulative gain and lift charts for model performance assessment in mineral potential mapping

Model performance assessment is a key procedure for mineral potential mapping, but the correspond-ing research achievements are seldom reported in literature. Cumulative gain and lift charts are well known in the data mining community specialized in marketing and sales applications and widely used in customer chum prediction for model performance assessment. In this paper, they are introduced into the field of mineral poten-tial mapping for model performance assessment. These two charts can be viewed as a graphic representation of the advantage of using a predictive model to choose mineral targets. A cumulative gain curve can represent how much a predictive model is superior to a random guess in mineral target prediction. A lift chart can express how much more likely the mineral targets predicted by a model are deposit-bearing ones than those by a random se-lection. As an illustration, the cumulative gain and lift charts are applied to measure the performance of weights of evidence, logistic regression,restricted Boltzmann machine, and multilayer perceptron in mineral potential mapping in the Altay district in northern Xinjiang in China. The results show that the cumulative gain and lift charts can visually reveal that the first three models perform well while the last one performs poorly. Thus, the cumulative gain and lift charts can serve as a graphic tool for model performance assessment in mineral potential mapping.

Velocity forecasts using a combined deep learning model in hybrid electric vehicles with V2V and V2I communication

Vehicle velocity forecast is an important clue in improving the performance of energy management in hybrid electric vehicles(HEV). This paper presents a new combined model for predicting vehicle’s velocity time series. The main features of the model are to combine the feature extraction capability of deep restricted Boltzmann machines(DBM) and sequence pattern predicting capability of bidirectional long short-term memory(BLSTM). Hence, the model is named as DBMBLSTM. In addition, the DRMBLSTM model utilizes the vehicle driving information and roadside infrastructure information provided respectively through vehicle-to-vehicle(V2V) and vehicle-to-infrastructure(V2I) communication channels to predict vehicle velocity at various length of prediction horizon. Furthermore, the predictions results of this study are compared with the state of the art of vehicle velocity forecasts. The root mean square error(RMSE) is used as an evaluation criteria of predictions accuracy. Finally,these compared prediction model are applied in model predictive control(MPC) energy management strategy for the verifications of fuel economy improvement of a HEV. Simulation results confirm that the proposed combined deep learning model performs better than other five prediction methods. Therefore, it is a means of arriving at a reliable forecast model for HEV.

Optimization of deep network models through fine tuning

Purpose–Many strategies have been put forward for training deep network models,however,stacking of several layers of non-linearities typically results in poor propagation of gradients and activations.The purpose of this paper is to explore the use of two steps strategy where initial deep learning model is obtained first by unsupervised learning and then optimizing the initial deep learning model by fine tuning.A number of fine tuning algorithms are explored in this work for optimizing deep learning models.This includes proposing a new algorithm where Backpropagation with adaptive gain algorithm is integrated with Dropout technique and the authors evaluate its performance in the fine tuning of the pretrained deep network.Design/methodology/approach–The parameters of deep neural networks are first learnt using greedy layer-wise unsupervised pretraining.The proposed technique is then used to perform supervised fine tuning of the deep neural network model.Extensive experimental study is performed to evaluate the performance of the proposed fine tuning technique on three benchmark data sets:USPS,Gisette and MNIST.The authors have tested the approach on varying size data sets which include randomly chosen training samples of size 20,50,70 and 100 percent from the original data set.Findings–Through extensive experimental study,it is concluded that the two steps strategy and the proposed fine tuning technique significantly yield promising results in optimization of deep network models.Originality/value–This paper proposes employing several algorithms for fine tuning of deep network model.A new approach that integrates adaptive gain Backpropagation(BP)algorithm with Dropout technique is proposed for fine tuning of deep networks.Evaluation and comparison of various algorithms proposed for fine tuning on three benchmark data sets is presented in the paper.