Classification of Conversational Sentences Using an Ensemble Pre-Trained Language Model with the Fine-Tuned Parameter

Sentence classification is the process of categorizing a sentence based on the context of the sentence.Sentence categorization requires more semantic highlights than other tasks,such as dependence parsing,which requires more syntactic elements.Most existing strategies focus on the general semantics of a conversation without involving the context of the sentence,recognizing the progress and comparing impacts.An ensemble pre-trained language model was taken up here to classify the conversation sentences from the conversation corpus.The conversational sentences are classified into four categories:information,question,directive,and commission.These classification label sequences are for analyzing the conversation progress and predicting the pecking order of the conversation.Ensemble of Bidirectional Encoder for Representation of Transformer(BERT),Robustly Optimized BERT pretraining Approach(RoBERTa),Generative Pre-Trained Transformer(GPT),DistilBERT and Generalized Autoregressive Pretraining for Language Understanding(XLNet)models are trained on conversation corpus with hyperparameters.Hyperparameter tuning approach is carried out for better performance on sentence classification.This Ensemble of Pre-trained Language Models with a Hyperparameter Tuning(EPLM-HT)system is trained on an annotated conversation dataset.The proposed approach outperformed compared to the base BERT,GPT,DistilBERT and XLNet transformer models.The proposed ensemble model with the fine-tuned parameters achieved an F1_score of 0.88.

Iris Liveness Detection Using Fragmental Energy of Haar Transformed Iris Images Using Ensemble of Machine Learning Classifiers

Contactless verification is possible with iris biometric identification,which helps prevent infections like COVID-19 from spreading.Biometric systems have grown unsteady and dangerous as a result of spoofing assaults employing contact lenses,replayed the video,and print attacks.The work demonstrates an iris liveness detection approach by utilizing fragmental coefficients of Haar transformed Iris images as signatures to prevent spoofing attacks for the very first time in the identification of iris liveness.Seven assorted feature creation ways are studied in the presented solutions,and these created features are explored for the training of eight distinct machine learning classifiers and ensembles.The predicted iris liveness identification variants are evaluated using recall,F-measure,precision,accuracy,APCER,BPCER,and ACER.Three standard datasets were used in the investigation.The main contribution of our study is achieving a good accuracy of 99.18%with a smaller feature vector.The fragmental coefficients of Haar transformed iris image of size 8∗8 utilizing random forest algorithm showed superior iris liveness detection with reduced featured vector size(64 features).Random forest gave 99.18%accuracy.Additionally,conduct an extensive experiment on cross datasets for detailed analysis.The results of our experiments showthat the iris biometric template is decreased in size tomake the proposed framework suitable for algorithmic verification in real-time environments and settings.

A Deep Learning Ensemble Method for Forecasting Daily Crude Oil Price Based on Snapshot Ensemble of Transformer Model

The oil industries are an important part of a country’s economy.The crude oil’s price is influenced by a wide range of variables.Therefore,how accurately can countries predict its behavior and what predictors to employ are two main questions.In this view,we propose utilizing deep learning and ensemble learning techniques to boost crude oil’s price forecasting performance.The suggested method is based on a deep learning snapshot ensemble method of the Transformer model.To examine the superiority of the proposed model,this paper compares the proposed deep learning ensemble model against different machine learning and statistical models for daily Organization of the Petroleum Exporting Countries(OPEC)oil price forecasting.Experimental results demonstrated the outperformance of the proposed method over statistical and machine learning methods.More precisely,the proposed snapshot ensemble of Transformer method achieved relative improvement in the forecasting performance compared to autoregressive integrated moving average ARIMA(1,1,1),ARIMA(0,1,1),autoregressive moving average(ARMA)(0,1),vector autoregression(VAR),random walk(RW),support vector machine(SVM),and random forests(RF)models by 99.94%,99.62%,99.87%,99.65%,7.55%,98.38%,and 99.35%,respectively,according to mean square error metric.

基于自适应噪声完全集合经验模态分解与BiLSTM-Transformer的锂离子电池剩余使用寿命预测

锂离子电池剩余使用寿命(remaining useful life,RUL)是使用者十分关心的问题,其涉及电池的更换时间和安全。针对锂离子电池的电容量非线性变化趋势,提出了一种基于自适应噪声完全集合经验模态分解与双向长短期记忆网络-Transformer的锂离子电池剩余使用寿命预测方法。首先,利用自适应噪声完全集合经验模态分解方法对锂离子电池电容量数据进行分解。其次,使用串联的双向长短期记忆神经网络和Transformer网络对分解后得到的残差序列和本征模态分量序列进行建模预测。最后,将预测的若干本征模态分量序列和残差序列进行求和,并对求和之后的最终预测数据与原始数据进行RUL预测。采用NASA公开的电池数据集对所提方法进行验证,结果表明,所提方法的平均绝对误差、均方根误差、平均绝对百分比误差和绝对误差控制分别控制在0.0173、0.0231、1.2084%和3个循环周期以内,能够有效地提高锂离子电池RUL的预测精度。

Impact of Perturbation Schemes on the Ensemble Prediction in a Coupled Lorenz Model

Based on a simple coupled Lorenz model,we investigate how to assess a suitable initial perturbation scheme for ensemble forecasting in a multiscale system involving slow dynamics and fast dynamics.Four initial perturbation approaches are used in the ensemble forecasting experiments:the random perturbation(RP),the bred vector(BV),the ensemble transform Kalman filter(ETKF),and the nonlinear local Lyapunov vector(NLLV)methods.Results show that,regardless of the method used,the ensemble averages behave indistinguishably from the control forecasts during the first few time steps.Due to different error growth in different time-scale systems,the ensemble averages perform better than the control forecast after very short lead times in a fast subsystem but after a relatively long period of time in a slow subsystem.Due to the coupled dynamic processes,the addition of perturbations to fast variables or to slow variables can contribute to an improvement in the forecasting skill for fast variables and slow variables.Regarding the initial perturbation approaches,the NLLVs show higher forecasting skill than the BVs or RPs overall.The NLLVs and ETKFs had nearly equivalent prediction skill,but NLLVs performed best by a narrow margin.In particular,when adding perturbations to slow variables,the independent perturbations(NLLVs and ETKFs)perform much better in ensemble prediction.These results are simply implied in a real coupled air–sea model.For the prediction of oceanic variables,using independent perturbations(NLLVs)and adding perturbations to oceanic variables are expected to result in better performance in the ensemble prediction.

Observation points classifier ensemble for high-dimensional imbalanced classification

In this paper,an Observation Points Classifier Ensemble(OPCE)algorithm is proposed to deal with High-Dimensional Imbalanced Classification(HDIC)problems based on data processed using the Multi-Dimensional Scaling(MDS)feature extraction technique.First,dimensionality of the original imbalanced data is reduced using MDS so that distances between any two different samples are preserved as well as possible.Second,a novel OPCE algorithm is applied to classify imbalanced samples by placing optimised observation points in a low-dimensional data space.Third,optimization of the observation point mappings is carried out to obtain a reliable assessment of the unknown samples.Exhaustive experiments have been conducted to evaluate the feasibility,rationality,and effectiveness of the proposed OPCE algorithm using seven benchmark HDIC data sets.Experimental results show that(1)the OPCE algorithm can be trained faster on low-dimensional imbalanced data than on high-dimensional data;(2)the OPCE algorithm can correctly identify samples as the number of optimised observation points is increased;and(3)statistical analysis reveals that OPCE yields better HDIC performances on the selected data sets in comparison with eight other HDIC algorithms.This demonstrates that OPCE is a viable algorithm to deal with HDIC problems.

Comparison of Nonlinear Local Lyapunov Vectors with Bred Vectors, Random Perturbations and Ensemble Transform Kalman Filter Strategies in a Barotropic Model

The breeding method has been widely used to generate ensemble perturbations in ensemble forecasting due to its simple concept and low computational cost. This method produces the fastest growing perturbation modes to catch the growing components in analysis errors. However, the bred vectors （BVs） are evolved on the same dynamical flow, which may increase the dependence of perturbations. In contrast, the nonlinear local Lyapunov vector （NLLV） scheme generates flow-dependent perturbations as in the breeding method, but regularly conducts the Gram-Schmidt reorthonormalization processes on the perturbations. The resulting NLLVs span the fast-growing perturbation subspace efficiently, and thus may grasp more com- ponents in analysis errors than the BVs. In this paper, the NLLVs are employed to generate initial ensemble perturbations in a barotropic quasi-geostrophic model. The performances of the ensemble forecasts of the NLLV method are systematically compared to those of the random pertur- bation （RP） technique, and the BV method, as well as its improved version--the ensemble transform Kalman filter （ETKF） method. The results demonstrate that the RP technique has the worst performance in ensemble forecasts, which indicates the importance of a flow-dependent initialization scheme. The ensemble perturbation subspaces of the NLLV and ETKF methods are preliminarily shown to catch similar components of analysis errors, which exceed that of the BVs. However, the NLLV scheme demonstrates slightly higher ensemble forecast skill than the ETKF scheme. In addition, the NLLV scheme involves a significantly simpler algorithm and less computation time than the ETKF method, and both demonstrate better ensemble forecast skill than the BV scheme.

Ensemble Transform Sensitivity Method for Adaptive Observations

The Ensemble Transform（ET） method has been shown to be useful in providing guidance for adaptive observation deployment.It predicts forecast error variance reduction for each possible deployment using its corresponding transformation matrix in an ensemble subspace.In this paper,a new ET-based sensitivity（ETS） method,which calculates the gradient of forecast error variance reduction in terms of analysis error variance reduction,is proposed to specify regions for possible adaptive observations.ETS is a first order approximation of the ET;it requires just one calculation of a transformation matrix,increasing computational efficiency（60%-80%reduction in computational cost）.An explicit mathematical formulation of the ETS gradient is derived and described.Both the ET and ETS methods are applied to the Hurricane Irene（2011） case and a heavy rainfall case for comparison.The numerical results imply that the sensitive areas estimated by the ETS and ET are similar.However,ETS is much more efficient,particularly when the resolution is higher and the number of ensemble members is larger.

Ensemble transform sensitivity for adaptive observations:a general formulation and its practicable implementation

The Ensemble Transformation(ET)method and its variation ET-based Sensitivity(ETS)method have been used in adaptive observation studies.However,the solution of the ensemble transformation matrix in the ET and ETS methods is not unique.A general mathematical formulation for the ensemble transformation matrix is derived and then a generalized equation for the ETS method is derived.It is proved that the previous ETS formulation is a special implementation of the newly derived general formulation.Another practicable implementation of the general ETS formulation that avoids calculating the inverse of some matrices is proposed.This ETS implementation showed physically reasonable statistical sensitivity regions for improving 1–3 day weather forecasts over eastern regions of U.S.A and Beijing region,China.

Classification of Adversarial Attacks Using Ensemble Clustering Approach

As more business transactions and information services have been implemented via communication networks,both personal and organization assets encounter a higher risk of attacks.To safeguard these,a perimeter defence likeNIDS(network-based intrusion detection system)can be effective for known intrusions.There has been a great deal of attention within the joint community of security and data science to improve machine-learning based NIDS such that it becomes more accurate for adversarial attacks,where obfuscation techniques are applied to disguise patterns of intrusive traffics.The current research focuses on non-payload connections at the TCP(transmission control protocol)stack level that is applicable to different network applications.In contrary to the wrapper method introduced with the benchmark dataset,three new filter models are proposed to transform the feature space without knowledge of class labels.These ECT(ensemble clustering based transformation)techniques,i.e.,ECT-Subspace,ECT-Noise and ECT-Combined,are developed using the concept of ensemble clustering and three different ensemble generation strategies,i.e.,random feature subspace,feature noise injection and their combinations.Based on the empirical study with published dataset and four classification algorithms,new models usually outperform that original wrapper and other filter alternatives found in the literature.This is similarly summarized from the first experiment with basic classification of legitimate and direct attacks,and the second that focuses on recognizing obfuscated intrusions.In addition,analysis of algorithmic parameters,i.e.,ensemble size and level of noise,is provided as a guideline for a practical use.

Ensemble-based optimization of hydraulically fractured horizontal well placement in shale gas reservoir through Hough transform parameterization

Shale gas reservoirs have been successfully developed due to the advancement of the horizontal well drilling and multistage hydraulic fracturing techniques.However,the optimization design of the horizontal well drilling,hydraulic fracturing,and operational schedule is a challenging problem.An ensemble-based optimization method(EnOpt)is proposed here to optimize the design of the hydraulically fractured horizontal well in the shale gas reservoir.The objective is to maximize the net present value(NPV)which requires a simulation model to predict the cumulative shale gas production.To accurately describe the geometry of the hydraulic fractures,the embedded discrete fracture modeling method(EDFM)is used to construct the shale gas simulation model.The efects of gas absorption,Knudsen difusion,natural and hydraulic fractures,and gas-water two phase fow are considered in the shale gas production system.To improve the parameter continuity and Gaussianity required by the EnOpt method,the Hough transformation parameterization is used to characterize the horizontal well.The results show that the proposed method can efectively optimize the design parameters of the hydraulically fractured horizontal well,and the NPV can be improved greatly after optimization so that the design parameters can approach to their optimal values.

集成卷积神经网络和视觉Transformer的隧道掌子面岩性判识研究

为研究综合高效的隧道掌子面岩性智能分类方法,首先,通过收集高铁沿线施工隧道高清掌子面照片、地质素描图及工程地质说明,筛选并统计出灰岩、泥岩、砂岩、玄武岩4种岩性,在此基础上,采用图像增强扩充样本数量并构建岩性样本集;然后,基于上述样本集分别构建ResNet50V2岩性分类迁移模型及VIT岩性分类模型,对比二者岩性分类效果,并采用Stacking方法集成2种模型的分类特点;最后,通过对比3种元学习器(逻辑回归、支持向量机、决策树)对2种模型的集成融合效果来选取最适用的元学习器。结果表明:采用逻辑回归集成ResNet50V2及VIT所构建的集成模型对岩性的分类效果最好,能充分融合掌子面岩性的全、局部特征来进行分类,模型准确率达到93.8%。

Price prediction of power transformer materials based on CEEMD and GRU

The rapid growth of the Chinese economy has fueled the expansion of power grids.Power transformers are key equipment in power grid projects,and their price changes have a significant impact on cost control.However,the prices of power transformer materials manifest as nonsmooth and nonlinear sequences.Hence,estimating the acquisition costs of power grid projects is difficult,hindering the normal operation of power engineering construction.To more accurately predict the price of power transformer materials,this study proposes a method based on complementary ensemble empirical mode decomposition(CEEMD)and gated recurrent unit(GRU)network.First,the CEEMD decomposed the price series into multiple intrinsic mode functions(IMFs).Multiple IMFs were clustered to obtain several aggregated sequences based on the sample entropy of each IMF.Then,an empirical wavelet transform(EWT)was applied to the aggregation sequence with a large sample entropy,and the multiple subsequences obtained from the decomposition were predicted by the GRU model.The GRU model was used to directly predict the aggregation sequences with a small sample entropy.In this study,we used authentic historical pricing data for power transformer materials to validate the proposed approach.The empirical findings demonstrated the efficacy of our method across both datasets,with mean absolute percentage errors(MAPEs)of less than 1%and 3%.This approach holds a significant reference value for future research in the field of power transformer material price prediction.

基于CEEMDAN-Transformer的灌浆流量混合预测模型

灌浆流量是最重要的水利工程灌浆参数之一,通过对灌浆流量的有效预测,可以实现对异常工况的提前响应,以保障施工质量与工程安全。然而由于灌浆过程面临的复杂地质情况,灌浆流量数据存在强非线性与波动性的特点,难以获得令人满意的计算精度。现有灌浆流量预测存在的不足如下:传统神经网络模型对时间序列特征提取和加工处理不足,导致预测精度有限;传统神经网络模型测试集进行一次计算仅能输出一个结果,进行多个时间步预测需要繁杂的多次计算;单测点预测结果预测时间短并且无法反映灌浆流量序列变化的整体趋势,不利于控制灌浆流量和保障施工质量。针对上述问题,本研究提出基于CEEMDAN-Transformer的灌浆流量混合预测模型。基于完全自适应噪声集合经验模态分解(Complete Ensemble Empirical Mode Decomposition with Adaptive Noise,CEEMDAN)方法将灌浆流量分解为本征模函数与残差信号,解决灌浆流量数据的非线性与强波动的问题;采用多头注意力Transformer实现多个本征模函数(Intrinsic Mode Function,IMF)序列到序列的预测,采用多头注意力机制来构建输入和输出的全局依赖关系,提升时间序列参数特征提取水平;最后,建立时序测点多输入多输出模型实现灌浆流量预测,提升多输出序列计算效率,反映整体趋势的多输出序列能够为灌浆流量控制提供参考。工程应用结果表明,本研究提出的基于CEEMDAN-Transformer的灌浆流量混合预测模型具有较好的计算精度和计算效率。

Practical implementation of Hilbert-Huang Transform algorithm

Hilbert-Huang Transform (HHT) is a newly developed powerful method for nonlinear and non-stationary time series analysis. The empirical mode decomposition is the key part of HHT, while its algorithm was protected by NASA as a US patent, which limits the wide application among the scientific community. Two approaches, mirror periodic and extrema extending methods, have been developed for handling the end effects of empirical mode decomposition. The implementation of the HHT is realized in detail to widen the application. The detailed comparison of the results from two methods with that from Huang et al. (1998, 1999), and the comparison between two methods are presented. Generally, both methods reproduce faithful results as those of Huang et al. For mirror periodic method (MPM), the data are extended once forever. Ideally, it is a way for handling the end effects of the HHT, especially for the signal that has symmetric waveform. The extrema extending method (EEM) behaves as good as MPM, and it is better than MPM for the signal that has strong asymmetric waveform. However, it has to perform extrema envelope extending in every shifting process.

AE waveform characteristics of rock mass under uniaxial loading based on Hilbert-Huang transform

Acoustic Emission(AE)waveforms contain information on microscopic structural features that can be related with damage of coal rock masses.In this paper,the Hilbert-Huang transform(HHT)method is used to obtain detailed structural characteristics of coal rock masses associated with damage,at different loading stages,from the analyses of the characteristics of AE waveforms.The results show that the HHT method can be used to decompose the target waveform into multiple intrinsic mode function(IMF)components,with the energy mainly concentrated in the c1−c4 IMF components,where the c1 component has the highest frequency and the largest amount of energy.As the loading continues,the proportion of energy occupied by the low-frequency IMF component shows an increasing trend.In the initial compaction stage,the Hilbert marginal spectrum is mainly concentrated in the low frequency range of 0−40 kHz.The plastic deformation stage is associated to energy accumulation in the frequency range of 0−25 kHz and 200−350 kHz,while the instability damage stage is mainly concentrated in the frequency range of 0−25 kHz.At 20 kHz,the instability damage reaches its maximum value.There is a relatively clear instantaneous energy peak at each stage,albeit being more distinct at the beginning and at the end of the compaction phase.Since the effective duration of the waveform is short,its resulting energy is small,and so there is a relatively high value from the instantaneous energy peak.The waveform lasts a relatively long time after the peak that coincides with failure,which is the period where the waveform reaches its maximum energy level.The Hilbert three-dimensional energy spectrum is generally zero in the region where the real energy is zero.In addition,its energy spectrum is intermittent rather than continuous.It is therefore consistent with the characteristics of the several dynamic ranges mentioned above,and it indicates more clearly the low-frequency energy concentration in the critical stage of instability failure.This study well reflects the response law of geophysical signals in the process of coal rock instability and failure,providing a basis for monitoring coal rock dynamic disasters.

Structural modal parameter identifi cation and damage diagnosis based on Hilbert-Huang transform

Traditional modal parameter identifi cation methods have many disadvantages,especially when used for processing nonlinear and non-stationary signals.In addition,they are usually not able to accurately identify the damping ratio and damage.In this study,methods based on the Hilbert-Huang transform(HHT) are investigated for structural modal parameter identifi cation and damage diagnosis.First,mirror extension and prediction via a radial basis function(RBF) neural network are used to restrain the troublesome end-effect issue in empirical mode decomposition(EMD),which is a crucial part of HHT.Then,the approaches based on HHT combined with other techniques,such as the random decrement technique(RDT),natural excitation technique(NExT) and stochastic subspace identifi cation(SSI),are proposed to identify modal parameters of structures.Furthermore,a damage diagnosis method based on the HHT is also proposed.Time-varying instantaneous frequency and instantaneous energy are used to identify the damage evolution of the structure.The relative amplitude of the Hilbert marginal spectrum is used to identify the damage location of the structure.Finally,acceleration records at gauge points from shaking table testing of a 12-story reinforced concrete frame model are taken to validate the proposed approaches.The results show that the proposed approaches based on HHT for modal parameter identifi cation and damage diagnosis are reliable and practical.

A technique to improve the empirical mode decomposition in the Hilbert-Huang transform

The Hilbert-based time-frequency analysis has promising capacity to reveal the time-variant behaviors of a sys- tem.To admit well-behaved Hilbert transforms,component decomposition of signals must be performed beforehand.This was first systematically implemented by the empirical mode decomposition(EMD)in the Hilbert-Huang transform,which can provide a time-frequency representation of the signals.The EMD,however,has limitations in distinguishing different components in narrowband signals commonly found in free-decay vibration signals.In this study,a technique for decompo- sing components in narrowband signals based on waves' beating phenomena is proposed to improve the EMD,in which the time scale structure of the signal is unveiled by the Hilbert transform as a result of wave beating,the order of component ex- traction is reversed from that in the EMD and the end effect is confined.The proposed technique is verified by performing the component decomposition of a simulated signal and a free decay signal actually measured in an instrumented bridge structure.In addition,the adaptability of the technique to time-variant dynamic systems is demonstrated with a simulated time-variant MDOF system.

Application of Hilbert-Huang transform to denoising in vortex flowmeter

Due to piping vibration, fluid pulsation and other environmental disturbances, variations of amplitude and frequency to the raw signals of vortex flowmeter are imposed. It is difficult to extract vortex frequencies which indicate volumetric flowrate from noisy data, especially at low flowrates. Hilbert-Huang transform was adopted to estimate vortex frequency. The noisy raw signal was decomposed into different intrinsic modes by empirical mode decomposition, the time-frequency characteristics of each mode were analyzed, and the vortex frequency was obtained by calculating partial mode’s instantaneous frequency. Experimental results show that the proposed method can estimate the vortex frequency with less than 2% relative error; and in the low flowrate range studied, the denoising ability of Hilbert-Huang transform is markedly better than Fourier based algorithms. These findings reveal that this method is accurate for vortex signal processing and at the same time has strong anti-disturbance ability.

The Effects of Land Surface Process Perturbations in a Global Ensemble Forecast System

Atmospheric variability is driven not only by internal dynamics, but also by external forcing, such as soil states, SST, snow, sea-ice cover, and so on. To investigate the forecast uncertainties and effects of land surface processes on numerical weather prediction, we added modules to perturb soil moisture and soil temperature into NCEP＇s Global Ensemble Forecast System （GEFS）, and compared the results of a set of experiments involving different configurations of land surface and atmospheric perturbation. It was found that uncertainties in different soil layers varied due to the multiple timescales of interactions between land surface and atmospheric processes. Perturbations of the soil moisture and soil temperature at the land surface changed sensible and latent heat flux obviously, as compared to the less or indirect land surface perturbation experiment from the day-to-day forecasts. Soil state perturbations led to greater variation in surface heat fluxes that transferred to the upper troposphere, thus reflecting interactions and the response to atmospheric external forcing. Various verification scores were calculated in this study. The results indicated that taking the uncertainties of land surface processes into account in GEFS could contribute a slight improvement in forecast skill in terms of resolution and reliability, a noticeable reduction in forecast error, as well as an increase in ensemble spread in an under-dispersive system. This paper provides a preliminary evaluation of the effects of land surface processes on predictability. Further research using more complex and suitable methods is needed to fully explore our understanding in this area.