Generalization properties of restricted Boltzmann machine for short-range order

A biased sampling algorithm for the restricted Boltzmann machine(RBM) is proposed, which allows generating configurations with a conserved quantity. To validate the method, a study of the short-range order in binary alloys with positive and negative exchange interactions is carried out. The network is trained on the data collected by Monte–Carlo simulations for a simple Ising-like binary alloy model and used to calculate the Warren–Cowley short-range order parameter and other thermodynamic properties. We demonstrate that the proposed method allows us not only to correctly reproduce the order parameters for the alloy concentration at which the network was trained, but can also predict them for any other concentrations.

Restricted Boltzmann machine: Recent advances and mean-field theory

This review deals with restricted Boltzmann machine(RBM) under the light of statistical physics.The RBM is a classical family of machine learning(ML) models which played a central role in the development of deep learning.Viewing it as a spin glass model and exhibiting various links with other models of statistical physics,we gather recent results dealing with mean-field theory in this context.First the functioning of the RBM can be analyzed via the phase diagrams obtained for various statistical ensembles of RBM,leading in particular to identify a compositional phase where a small number of features or modes are combined to form complex patterns.Then we discuss recent works either able to devise mean-field based learning algorithms;either able to reproduce generic aspects of the learning process from some ensemble dynamics equations or/and from linear stability arguments.

FPGA Implementation of a Scalable and Highly Parallel Architecture for Restricted Boltzmann Machines

Restricted Boltzmann Machines (RBMs) are an effective model for machine learning;however, they require a significant amount of processing time. In this study, we propose a highly parallel, highly flexible architecture that combines small and completely parallel RBMs. This proposal addresses problems associated with calculation speed and exponential increases in circuit scale. We show that this architecture can optionally respond to the trade-offs between these two problems. Furthermore, our FPGA implementation performs at a 134 times processing speed up factor with respect to a conventional CPU.

Sustainable Investment Forecasting of Power Grids Based on theDeep Restricted Boltzmann Machine Optimized by the Lion Algorithm

This paper proposes a new power grid investment prediction model based on the deep restricted Boltzmann machine(DRBM)optimized by the Lion algorithm(LA).Firstly,two factors including transmission and distribution price reform(TDPR)and 5G station construction were comprehensively incorporated into the consideration of influencing factors,and the fuzzy threshold method was used to screen out critical influencing factors.Then,the LA was used to optimize the parameters of the DRBM model to improve the model’s prediction accuracy,and the model was trained with the selected influencing factors and investment.Finally,the LA-DRBM model was used to predict the investment of a power grid enterprise,and the final prediction result was obtained by modifying the initial result with the modifying factors.The LA-DRBMmodel compensates for the deficiency of the singlemodel,and greatly improves the investment prediction accuracy of the power grid.In this study,a power grid enterprise was taken as an example to carry out an empirical analysis to prove the validity of the model,and a comparison with the RBM,support vector machine(SVM),back propagation neural network(BPNN),and regression model was conducted to verify the superiority of the model.The conclusion indicates that the proposed model has a strong generalization ability and good robustness,is able to abstract the combination of low-level features into high-level features,and can improve the efficiency of the model’s calculations for investment prediction of power grid enterprises.

Classification of steel based on laser-induced breakdown spectroscopy combined with restricted Boltzmann machine and support vector machine

In recent years,a laser-induced breakdown spectrometer(LIBS)combined with machine learning has been widely developed for steel classification.However,the much redundant information of LIBS spectra increases the computation complexity for classification.In this work,restricted Boltzmann machines(RBM)and principal component analysis(PCA)were used for dimension reduction of datasets,respectively.Then,a support vector machine(SVM)was adopted to process feature information.Two models(RBM-SVM and PCA-SVM)are compared in terms of performance.After optimization,the accuracy of the RBM-SVM model can achieve 100%,and the maximum dimension reduction time is 33.18 s,which is nearly half of that of the PCA model(53.19 s).These results preliminarily indicate that LIBS combined with RBM-SVM has great potential in the real-time classification of steel.

基于MPSR和IRBM的电力系统中长期负荷预测

针对电力系统中长期负荷波动大及不确定因素导致负荷预测误差较大的问题,提出了一种基于多变量相空间重构(MPSR)和改进受限波尔兹曼机(IRBM)的电力系统中长期负荷预测方法。首先,利用多元线性回归分析方法分析天气因素与电负荷之间的相关性,并将其与电负荷序列组成多变量时间序列;然后,利用C-C法确定每一时间序列的最优嵌入维数和时间延迟,实现多变量相空间重构;最后,采用多变量相空间重构建立的数据集训练电力系统负荷预测模型,同时利用梯度优化法对参数进行优化,得到预测模型。结果表明:相比长短期记忆神经网络和粒子群优化BP神经网络,所提出的预测方法有较高的精准度。

Green's function Monte Carlo method combined with restricted Boltzmann machine approach to the frustrated J_(1)–J_(2)Heisenberg model

Restricted Boltzmann machine(RBM)has been proposed as a powerful variational ansatz to represent the ground state of a given quantum many-body system.On the other hand,as a shallow neural network,it is found that the RBM is still hardly able to capture the characteristics of systems with large sizes or complicated interactions.In order to find a way out of the dilemma,here,we propose to adopt the Green's function Monte Carlo(GFMC)method for which the RBM is used as a guiding wave function.To demonstrate the implementation and effectiveness of the proposal,we have applied the proposal to study the frustrated J_(1)-J_(2)Heisenberg model on a square lattice,which is considered as a typical model with sign problem for quantum Monte Carlo simulations.The calculation results demonstrate that the GFMC method can significantly further reduce the relative error of the ground-state energy on the basis of the RBM variational results.This encourages to combine the GFMC method with other neural networks like convolutional neural networks for dealing with more models with sign problem in the future.

基于Lorentz函数的稀疏约束RBM模型的算法研究

受限玻尔兹曼机(Restricted Boltzmann Machine,RBM)是一种有效的特征提取算法,受视觉皮层稀疏表示的启发,人们试图将稀疏这一概念引入到RBM中,以期学习到原始数据的稀疏表示,提高其特征提取性能。将Lorentz函数引入到RBM中,作为RBM的稀疏约束正则项,构建基于Lorentz函数的稀疏约束RBM模型,将其称之为LRBM模型。对该模型的特征提取性能进行了可视化评价,同时对稀疏度和分类率进行了实验分析;最后将多个LRBM叠加,构造基于LRBM的深度置信网模型并分析该深度网络的性能。实验表明,LRBM模型有效地提取了数据集中的特征信息,在分类效果上较RBM平均提高了2%左右,增强了目标分类的可靠性。

基于深度学习和GB-RBM的UAV红外语义分割方法

为提高UAV红外图像语义分割的性能,提出基于深度学习和高斯伯努利受限玻尔兹曼机(GB-RBM)的实时语义分割模型。确认地面车辆实时特征提取中的关键问题。基于GB-RBM,提出用于编码阶段的形状先验模型。通过将SegNet中的编码器-解码器结构与GB-RBM模块相融合,在解码器块中生成红外数据的实时映射,实现准确快速的语义分割。实验结果表明,所提方法能够很好地处理红外视频中的实时几何信息,在3个实验数据集上的平均精度约为0.98,平均处理时长约为17.86 s,性能优于其它优秀方法。

结合RBM的MLP神经网络输变电工程量评估方法

为了解决输变电工程中工程量合理性的智能评估问题,该文提出一种结合RBM(玻尔兹曼机)的MLP(多层感知机)神经网络模型.该模型通过学习可信历史数据中影响因素和工程量的关系,具备了从影响因素预测工程量的能力;再通过对真实值与预测值之间差异的判断,自动评估目标工程量的合理性.为了能够让模型更好地从复杂的历史数据中学习,从而有效地提高MLP神经网络模型预测的精准度,文中引入玻尔兹曼机对历史数据进行无监督学习,提取可以表征原数据的新的抽象特征.仿真表明,该文方法能够有效推动输变电工程量的智能评估,解决目前专家人工评估中主观因素带来的问题.

基于CRBM算法的时间序列预测模型研究

针对受限玻尔兹曼机(restricted Boltzmann machines,RBM)算法对时序数据预测存在抽取抽象特征向量能力较差和梯度下降能力有限的问题,基于CRBM(conditional restricted Boltzmann machines)算法以及信念网络(deep belief network,DBN)模型,构建了一种非线性的CRBM-DBN深度学习模型,并采用高斯分布处理输入特征值和对比散度抽样,用于预测时序数据.实验以浙江省近岸海域赤潮时序数据作为输入特征值,讨论该模型的深度及参数选取,并与经典的深度学习模型RBM、DAE和浅层学习中的BP神经网络进行对比,实验验证CRBM对于赤潮时序数据的预测拟合度要明显优于其他3种模型,该模型可有效用于赤潮类时序数据的趋势性预测.

Monitoring and diagnosis of complex production process based on free energy of Gaussian–Bernoulli restricted Boltzmann machine

Online monitoring and diagnosis of production processes face great challenges due to the nonlinearity and multivariate of complex industrial processes.Traditional process monitoring methods employ kernel function or multilayer neural networks to solve the nonlinear mapping problem of data.However,the above methods increase the model complexity and are not interpretable,leading to difficulties in subsequent fault recognition/diagnosis/location.A process monitoring and diagnosis method based on the free energy of Gaussian-Bernoulli restricted Boltzmann machine(GBRBM-FE)was proposed.Firstly,a GBRBM network was established to make the probability distribution of the reconstructed data as close as possible to the probability distribution of the raw data.On this basis,the weights and biases in GBRBM network were used to construct F statistics,which represents the free energy of the sample.The smaller the energy of the sample is,the more normal the sample is.Therefore,F statistics can be used to monitor the production process.To diagnose fault variables,the F statistic for each sample was decomposed to obtain the Fv statistic for each variable.By analyzing the deviation degree between the corresponding variables of abnormal samples and normal samples,the cause of process abnormalities can be accurately located.The application of converter steelmaking process demonstrates that the proposed method outperforms the traditional methods,in terms of fault monitoring and diagnosis performance.

一种基于特定频段信息熵和RBM的健康因子构建方法

针对传统物理健康因子存在单调性差、对早期故障不敏感等问题,提出一种基于特定频段信息熵和受限玻尔兹曼机(SEI-RBM)的虚拟健康因子构建模型。该模型由物理健康因子构建层和特征融合层两部分组成:在物理健康因子构建层中,提出一种基于特定频段信息熵的物理健康因子构建方法;特征融合层中,利用单调性准则选取部分物理健康因子组成特征集,利用受限玻尔兹曼机(RBM)对健康因子特征集进行融合,得到虚拟健康因子。实验结果表明:利用该模型构建的虚拟健康因子能够有效提高滚动轴承性能退化曲线的单调性,有助于提高剩余寿命预测的精确度。

基于并行回火改进的GRBM的语音识别

为提高连续语音识别中的识别准确率,采用高斯伯努利受限玻尔兹曼机进行语音训练和识别。通过结合并行回火算法的思想,采样、交换不同的温度链下的重构数据,实现在全局范围内对整个分布进行采样,提出一种基于并行回火改进的高斯伯努利受限玻尔兹曼机(GRBM-PT)的建模方法。该方法通过对语音信号的连续数据进行预训练分析、建模,最后使用支持向量机作为语音识别的分类器。在TI-Digits数字语音训练和数字测试数据库上的实验结果表明,语音识别率能够达到83.14%,基于GRBM-PT模型下的语音识别率明显优于RBM,RBM-PT以及GRBM模型的性能。

基于动态Gibbs采样的RBM训练算法研究

目前大部分受限玻尔兹曼机(Restricted Boltzmann machines,RBMs)训练算法都是以多步Gibbs采样为基础的采样算法.本文针对多步Gibbs采样过程中出现的采样发散和训练速度过慢的问题,首先,对问题进行实验描述,给出了问题的具体形式;然后,从马尔科夫采样的角度对多步Gibbs采样的收敛性质进行了理论分析,证明了多步Gibbs采样在受限玻尔兹曼机训练初期较差的收敛性质是造成采样发散和训练速度过慢的主要原因;最后,提出了动态Gibbs采样算法,给出了对比仿真实验.实验结果表明,动态Gibbs采样算法可以有效地克服采样发散的问题,并且能够以微小的运行时间为代价获得更高的训练精度.

基于权值动量的RBM加速学习算法研究

动量算法理论上可以加速受限玻尔兹曼机(Restricted Boltzmann machine,RBM)网络的训练速度.本文通过对现有动量算法进行仿真研究,发现现有动量算法在受限玻尔兹曼机网络训练中加速效果较差,且在训练后期逐渐失去了加速性能.针对以上问题,本文首先基于Gibbs采样收敛性定理对现有动量算法进行了理论分析,证明了现有动量算法的加速效果是以牺牲网络权值为代价的;然后,本文进一步对网络权值进行研究,发现网络权值中包含大量真实梯度的方向信息,这些方向信息可以用来对网络进行训练;基于此,本文提出了基于网络权值的权值动量算法,最后给出了仿真实验.实验结果表明,本文提出的动量算法具有更好的加速效果,并且在训练后期仍然能够保持较好的加速性能,可以很好地弥补现有动量算法的不足.

基于RNN-RBM语言模型的语音识别研究

近年来深度学习兴起,其在语言模型领域有着不错的成效,如受限玻尔兹曼机(restricted Boltzmann machine,RBM)语言模型等.不同于N-gram语言模型,这些根植于神经网络的语言模型可以将词序列映射到连续空间来评估下一词出现的概率,以解决数据稀疏的问题.此外,也有学者使用递归神经网络来建构语言模型,期望由递归的方式充分利用所有上文信息来预测下一词,进而有效处理长距离语言约束.根据递归受限玻尔兹曼机神经网络(recurrent neural network-restricted Boltzmann machine,RNN-RBM)的基础来捕捉长距离信息;另外,也探讨了根据语言中语句的特性来动态地调整语言模型.实验结果显示,使用RNN-RBM语言模型对于大词汇连续语音识别的效能有相当程度的提升.

基于改进并行回火算法的RBM网络训练研究

目前受限玻尔兹曼机网络训练算法主要是基于采样的算法.当用采样算法进行梯度计算时,得到的采样梯度是真实梯度的近似值,采样梯度和真实梯度之间存在较大的误差,这严重影响了网络的训练效果.针对该问题,本文首先分析了采样梯度和真实梯度之间的数值误差和方向误差,以及它们对网络训练性能的影响,然后从马尔科夫采样的角度对以上问题进行了理论分析,并建立了梯度修正模型,通过修正梯度对采样梯度进行数值和方向的调节,并提出了基于改进并行回火算法的训练算法,即GFPT(Gradient fixing parallel tempering)算法.最后给出GFPT算法与现有算法的对比实验,仿真结果表明,GFPT算法可以极大地减小采样梯度和真实梯度之间的误差,大幅度提升受限玻尔兹曼机网络的训练效果.

基于遗传算法的RBM优化设计

为了有效解决受限玻尔兹曼机在设计时没有规律遵循并很难保证网络最优化的问题,提出一种基于遗传算法的RBM辅助优化设计方法 (Genetic Algorithm-Restricted Boltzmann Machine,GA-RBM),完成了RBM模型结构和权值的全局搜索.针对RBM特点,设计RBM模型个体编码方式和适应度函数,实现了通过遗传算法对可见单元维度的优化和隐单元个数的选择.通过MNIST实验证明,相较于其他常规的数据降维方式,该方法不仅可以降低可见单元维数,而且能够有效提高RBM特征提取性能,达到了通过遗传算法实现RBM模型优化设计的目的.

基于DRBM和边缘检测的脑部磁共振图像分类

为提高图像分类精度,文中提出了一种DRBM网络结合边缘检测的图像分类方法。该方法首先进行边缘检测,然后将提取的纹理叠加到原图中,实现图像纹理加强。接着构建可视层-隐层-分类器三层DRBM,实现特征提取并分类。实验证明,相比较传统基于单一特征的分类方法,文中方法取得了较高的分类准确率,具有更好的图像分类性能。