Generalization properties of restricted Boltzmann machine for short-range order

A biased sampling algorithm for the restricted Boltzmann machine(RBM) is proposed, which allows generating configurations with a conserved quantity. To validate the method, a study of the short-range order in binary alloys with positive and negative exchange interactions is carried out. The network is trained on the data collected by Monte–Carlo simulations for a simple Ising-like binary alloy model and used to calculate the Warren–Cowley short-range order parameter and other thermodynamic properties. We demonstrate that the proposed method allows us not only to correctly reproduce the order parameters for the alloy concentration at which the network was trained, but can also predict them for any other concentrations.

Restricted Boltzmann machine: Recent advances and mean-field theory

This review deals with restricted Boltzmann machine(RBM) under the light of statistical physics.The RBM is a classical family of machine learning(ML) models which played a central role in the development of deep learning.Viewing it as a spin glass model and exhibiting various links with other models of statistical physics,we gather recent results dealing with mean-field theory in this context.First the functioning of the RBM can be analyzed via the phase diagrams obtained for various statistical ensembles of RBM,leading in particular to identify a compositional phase where a small number of features or modes are combined to form complex patterns.Then we discuss recent works either able to devise mean-field based learning algorithms;either able to reproduce generic aspects of the learning process from some ensemble dynamics equations or/and from linear stability arguments.

Sustainable Investment Forecasting of Power Grids Based on theDeep Restricted Boltzmann Machine Optimized by the Lion Algorithm

This paper proposes a new power grid investment prediction model based on the deep restricted Boltzmann machine(DRBM)optimized by the Lion algorithm(LA).Firstly,two factors including transmission and distribution price reform(TDPR)and 5G station construction were comprehensively incorporated into the consideration of influencing factors,and the fuzzy threshold method was used to screen out critical influencing factors.Then,the LA was used to optimize the parameters of the DRBM model to improve the model’s prediction accuracy,and the model was trained with the selected influencing factors and investment.Finally,the LA-DRBM model was used to predict the investment of a power grid enterprise,and the final prediction result was obtained by modifying the initial result with the modifying factors.The LA-DRBMmodel compensates for the deficiency of the singlemodel,and greatly improves the investment prediction accuracy of the power grid.In this study,a power grid enterprise was taken as an example to carry out an empirical analysis to prove the validity of the model,and a comparison with the RBM,support vector machine(SVM),back propagation neural network(BPNN),and regression model was conducted to verify the superiority of the model.The conclusion indicates that the proposed model has a strong generalization ability and good robustness,is able to abstract the combination of low-level features into high-level features,and can improve the efficiency of the model’s calculations for investment prediction of power grid enterprises.

Classification of steel based on laser-induced breakdown spectroscopy combined with restricted Boltzmann machine and support vector machine

In recent years,a laser-induced breakdown spectrometer(LIBS)combined with machine learning has been widely developed for steel classification.However,the much redundant information of LIBS spectra increases the computation complexity for classification.In this work,restricted Boltzmann machines(RBM)and principal component analysis(PCA)were used for dimension reduction of datasets,respectively.Then,a support vector machine(SVM)was adopted to process feature information.Two models(RBM-SVM and PCA-SVM)are compared in terms of performance.After optimization,the accuracy of the RBM-SVM model can achieve 100%,and the maximum dimension reduction time is 33.18 s,which is nearly half of that of the PCA model(53.19 s).These results preliminarily indicate that LIBS combined with RBM-SVM has great potential in the real-time classification of steel.

Green's function Monte Carlo method combined with restricted Boltzmann machine approach to the frustrated J_(1)–J_(2)Heisenberg model

Restricted Boltzmann machine(RBM)has been proposed as a powerful variational ansatz to represent the ground state of a given quantum many-body system.On the other hand,as a shallow neural network,it is found that the RBM is still hardly able to capture the characteristics of systems with large sizes or complicated interactions.In order to find a way out of the dilemma,here,we propose to adopt the Green's function Monte Carlo(GFMC)method for which the RBM is used as a guiding wave function.To demonstrate the implementation and effectiveness of the proposal,we have applied the proposal to study the frustrated J_(1)-J_(2)Heisenberg model on a square lattice,which is considered as a typical model with sign problem for quantum Monte Carlo simulations.The calculation results demonstrate that the GFMC method can significantly further reduce the relative error of the ground-state energy on the basis of the RBM variational results.This encourages to combine the GFMC method with other neural networks like convolutional neural networks for dealing with more models with sign problem in the future.

FPGA Implementation of a Scalable and Highly Parallel Architecture for Restricted Boltzmann Machines

Restricted Boltzmann Machines (RBMs) are an effective model for machine learning;however, they require a significant amount of processing time. In this study, we propose a highly parallel, highly flexible architecture that combines small and completely parallel RBMs. This proposal addresses problems associated with calculation speed and exponential increases in circuit scale. We show that this architecture can optionally respond to the trade-offs between these two problems. Furthermore, our FPGA implementation performs at a 134 times processing speed up factor with respect to a conventional CPU.

基于MPSR和IRBM的电力系统中长期负荷预测

针对电力系统中长期负荷波动大及不确定因素导致负荷预测误差较大的问题,提出了一种基于多变量相空间重构(MPSR)和改进受限波尔兹曼机(IRBM)的电力系统中长期负荷预测方法。首先,利用多元线性回归分析方法分析天气因素与电负荷之间的相关性,并将其与电负荷序列组成多变量时间序列;然后,利用C-C法确定每一时间序列的最优嵌入维数和时间延迟,实现多变量相空间重构;最后,采用多变量相空间重构建立的数据集训练电力系统负荷预测模型,同时利用梯度优化法对参数进行优化,得到预测模型。结果表明:相比长短期记忆神经网络和粒子群优化BP神经网络,所提出的预测方法有较高的精准度。

Monitoring and diagnosis of complex production process based on free energy of Gaussian–Bernoulli restricted Boltzmann machine

Online monitoring and diagnosis of production processes face great challenges due to the nonlinearity and multivariate of complex industrial processes.Traditional process monitoring methods employ kernel function or multilayer neural networks to solve the nonlinear mapping problem of data.However,the above methods increase the model complexity and are not interpretable,leading to difficulties in subsequent fault recognition/diagnosis/location.A process monitoring and diagnosis method based on the free energy of Gaussian-Bernoulli restricted Boltzmann machine(GBRBM-FE)was proposed.Firstly,a GBRBM network was established to make the probability distribution of the reconstructed data as close as possible to the probability distribution of the raw data.On this basis,the weights and biases in GBRBM network were used to construct F statistics,which represents the free energy of the sample.The smaller the energy of the sample is,the more normal the sample is.Therefore,F statistics can be used to monitor the production process.To diagnose fault variables,the F statistic for each sample was decomposed to obtain the Fv statistic for each variable.By analyzing the deviation degree between the corresponding variables of abnormal samples and normal samples,the cause of process abnormalities can be accurately located.The application of converter steelmaking process demonstrates that the proposed method outperforms the traditional methods,in terms of fault monitoring and diagnosis performance.

基于深度学习和GB-RBM的UAV红外语义分割方法

为提高UAV红外图像语义分割的性能,提出基于深度学习和高斯伯努利受限玻尔兹曼机(GB-RBM)的实时语义分割模型。确认地面车辆实时特征提取中的关键问题。基于GB-RBM,提出用于编码阶段的形状先验模型。通过将SegNet中的编码器-解码器结构与GB-RBM模块相融合,在解码器块中生成红外数据的实时映射,实现准确快速的语义分割。实验结果表明,所提方法能够很好地处理红外视频中的实时几何信息,在3个实验数据集上的平均精度约为0.98,平均处理时长约为17.86 s,性能优于其它优秀方法。

AN ENSEMBLE MODEL OF ARIMA AND ANN WITH RESTRICTED BOLTZMANN MACHINE BASED ON DECOMPOSITION OF DISCRETE WAVELET TRANSFORM FOR TIME SERIES FORECASTING

Time series forecasting research area mainly focuses on developing effective forecasting models toimprove prediction accuracy. An ensemble model composed of autoregressive integrated movingaverage （ARIMA）, artificial neural network （ANN）, restricted Boltzmann machines （RBM）, anddiscrete wavelet transform （DWT） is presented in this paper. In the proposed model, DWT firstdecomposes time series into approximation and detail. Then Khashei and Bijari＇s model, which is anensemble model of ARIMA and ANN, is applied to the approximation and detail to extract their bothlinear and nonlinear components and fit the relationship between the components as a function insteadof additive relationship. Furthermore, RBM is used to perform pre-training for generating initialweights and biases based on inputs feature for ANN. Finally, the forecasted approximation and detailare combined to obtain final forecasting. The forecasting capability of the proposed model is testedwith three well-known time series： sunspot, Canadian lynx, exchange rate time series. The predictionperformance is compared to the other six forecasting models. The results indicate that the proposedmodel gives the best performance in all three data sets and all three measures （i.e. MSE, MAE andMAPE）.

结合RBM的MLP神经网络输变电工程量评估方法

为了解决输变电工程中工程量合理性的智能评估问题,该文提出一种结合RBM(玻尔兹曼机)的MLP(多层感知机)神经网络模型.该模型通过学习可信历史数据中影响因素和工程量的关系,具备了从影响因素预测工程量的能力;再通过对真实值与预测值之间差异的判断,自动评估目标工程量的合理性.为了能够让模型更好地从复杂的历史数据中学习,从而有效地提高MLP神经网络模型预测的精准度,文中引入玻尔兹曼机对历史数据进行无监督学习,提取可以表征原数据的新的抽象特征.仿真表明,该文方法能够有效推动输变电工程量的智能评估,解决目前专家人工评估中主观因素带来的问题.

A new item-based deep network structure using a restricted Boltzmann machine for collaborative filtering

The collaborative filtering(CF) technique has been widely used recently in recommendation systems. It needs historical data to give predictions. However, the data sparsity problem still exists. We propose a new item-based restricted Boltzmann machine(RBM) approach for CF and use the deep multilayer RBM network structure, which alleviates the data sparsity problem and has excellent ability to extract features. Each item is treated as a single RBM, and different items share the same weights and biases. The parameters are learned layer by layer in the deep network. The batch gradient descent algorithm with minibatch is used to increase the convergence speed. The new feature vector discovered by the multilayer RBM network structure is very effective in predicting a rating and achieves a better result. Experimental results on the data set of MovieL ens show that the item-based multilayer RBM approach achieves the best performance, with a mean absolute error of 0.6424 and a root-mean-square error of 0.7843.

受限玻尔兹曼机及其变体研究综述

受限玻尔兹曼机作为学习数据分布和提取内在特征的典型概率图模型,是深度学习领域重要的基础模型。近年来,通过改进受限玻尔兹曼机的模型结构和能量函数得到众多新兴模型,即受限玻尔兹曼机变体,可以进一步提升模型的特征提取性能。研究受限玻尔兹曼机及其变体能够显著促进深度学习领域的发展,实现大数据时代海量信息的快速提取。基于此,对近年来受限玻尔兹曼机及其变体的相关研究进行系统回顾,并创新性地从训练算法改进、模型结构改进、模型深层融合研究和模型相关最新应用4个方面进行全面综述。其中,重点梳理受限玻尔兹曼机训练算法和变体模型的发展史。最后,讨论受限玻尔兹曼机及其变体领域的现存难点与挑战,对主要研究工作进行总结与展望。

基于动态Gibbs采样的RBM训练算法研究

目前大部分受限玻尔兹曼机(Restricted Boltzmann machines,RBMs)训练算法都是以多步Gibbs采样为基础的采样算法.本文针对多步Gibbs采样过程中出现的采样发散和训练速度过慢的问题,首先,对问题进行实验描述,给出了问题的具体形式;然后,从马尔科夫采样的角度对多步Gibbs采样的收敛性质进行了理论分析,证明了多步Gibbs采样在受限玻尔兹曼机训练初期较差的收敛性质是造成采样发散和训练速度过慢的主要原因;最后,提出了动态Gibbs采样算法,给出了对比仿真实验.实验结果表明,动态Gibbs采样算法可以有效地克服采样发散的问题,并且能够以微小的运行时间为代价获得更高的训练精度.

基于权值动量的RBM加速学习算法研究

动量算法理论上可以加速受限玻尔兹曼机(Restricted Boltzmann machine,RBM)网络的训练速度.本文通过对现有动量算法进行仿真研究,发现现有动量算法在受限玻尔兹曼机网络训练中加速效果较差,且在训练后期逐渐失去了加速性能.针对以上问题,本文首先基于Gibbs采样收敛性定理对现有动量算法进行了理论分析,证明了现有动量算法的加速效果是以牺牲网络权值为代价的;然后,本文进一步对网络权值进行研究,发现网络权值中包含大量真实梯度的方向信息,这些方向信息可以用来对网络进行训练;基于此,本文提出了基于网络权值的权值动量算法,最后给出了仿真实验.实验结果表明,本文提出的动量算法具有更好的加速效果,并且在训练后期仍然能够保持较好的加速性能,可以很好地弥补现有动量算法的不足.

基于RNN-RBM语言模型的语音识别研究

近年来深度学习兴起,其在语言模型领域有着不错的成效,如受限玻尔兹曼机(restricted Boltzmann machine,RBM)语言模型等.不同于N-gram语言模型,这些根植于神经网络的语言模型可以将词序列映射到连续空间来评估下一词出现的概率,以解决数据稀疏的问题.此外,也有学者使用递归神经网络来建构语言模型,期望由递归的方式充分利用所有上文信息来预测下一词,进而有效处理长距离语言约束.根据递归受限玻尔兹曼机神经网络(recurrent neural network-restricted Boltzmann machine,RNN-RBM)的基础来捕捉长距离信息;另外,也探讨了根据语言中语句的特性来动态地调整语言模型.实验结果显示,使用RNN-RBM语言模型对于大词汇连续语音识别的效能有相当程度的提升.

基于改进并行回火算法的RBM网络训练研究

目前受限玻尔兹曼机网络训练算法主要是基于采样的算法.当用采样算法进行梯度计算时,得到的采样梯度是真实梯度的近似值,采样梯度和真实梯度之间存在较大的误差,这严重影响了网络的训练效果.针对该问题,本文首先分析了采样梯度和真实梯度之间的数值误差和方向误差,以及它们对网络训练性能的影响,然后从马尔科夫采样的角度对以上问题进行了理论分析,并建立了梯度修正模型,通过修正梯度对采样梯度进行数值和方向的调节,并提出了基于改进并行回火算法的训练算法,即GFPT(Gradient fixing parallel tempering)算法.最后给出GFPT算法与现有算法的对比实验,仿真结果表明,GFPT算法可以极大地减小采样梯度和真实梯度之间的误差,大幅度提升受限玻尔兹曼机网络的训练效果.

基于遗传算法的RBM优化设计

为了有效解决受限玻尔兹曼机在设计时没有规律遵循并很难保证网络最优化的问题,提出一种基于遗传算法的RBM辅助优化设计方法 (Genetic Algorithm-Restricted Boltzmann Machine,GA-RBM),完成了RBM模型结构和权值的全局搜索.针对RBM特点,设计RBM模型个体编码方式和适应度函数,实现了通过遗传算法对可见单元维度的优化和隐单元个数的选择.通过MNIST实验证明,相较于其他常规的数据降维方式,该方法不仅可以降低可见单元维数,而且能够有效提高RBM特征提取性能,达到了通过遗传算法实现RBM模型优化设计的目的.

基于判别性无穷模糊受限玻尔兹曼机模型的HRRP序列识别

针对雷达高分辨率距离像(High Resolution Range Profiles, HRRP)序列数据在受干扰或背景杂波严重时目标识别性能较差的问题,提出一种判别性无穷模糊受限玻尔兹曼机(Discriminative Infinite Fuzzy Re-stricted Boltzmann Machine, Dis-iFRBM)模型。该模型结合判别性受限玻尔兹曼机分类和无穷受限玻尔兹曼机模型复杂度自适应特点,汲取模糊神经网络在低信噪比环境下提取特征更稳定的优点,将模型参数从实数扩展为模糊参数,实现了对HRRP序列数据原始特征的更稳定提取以及对雷达目标的更稳健识别。通过对多个HRRP序列的识别实验,验证了Dis-iFRBM的识别稳定性以及鲁棒性,与其他模型的对比实验验证了所提模型在“噪声”环境中的有效性和优越性。

一种基于RBM的深层神经网络音素识别方法

为提高连续语音识别中的音素识别准确率,采用深可信网络提取语音音素后验概率进行音素识别。首先利用受限玻尔兹曼机的学习原理,对深可信网络进行逐层的预训练;然后通过增加一个"软最大化(softmax)"输出层,得到用于音素状态后验概率检测的深层神经网络,并采用后向传播算法进行网络权值的精细调整;最后以后验概率为HMM发射概率,使用Viterbi解码器进行音素识别。针对TIMIT语料库的实验结果表明,该系统的音素识别率优于GMM/HMM,MLP/HMM和TANDEM系统性能。