Non-crossing Quantile Regression Neural Network as a Calibration Tool for Ensemble Weather Forecasts

Despite the maturity of ensemble numerical weather prediction(NWP),the resulting forecasts are still,more often than not,under-dispersed.As such,forecast calibration tools have become popular.Among those tools,quantile regression(QR)is highly competitive in terms of both flexibility and predictive performance.Nevertheless,a long-standing problem of QR is quantile crossing,which greatly limits the interpretability of QR-calibrated forecasts.On this point,this study proposes a non-crossing quantile regression neural network(NCQRNN),for calibrating ensemble NWP forecasts into a set of reliable quantile forecasts without crossing.The overarching design principle of NCQRNN is to add on top of the conventional QRNN structure another hidden layer,which imposes a non-decreasing mapping between the combined output from nodes of the last hidden layer to the nodes of the output layer,through a triangular weight matrix with positive entries.The empirical part of the work considers a solar irradiance case study,in which four years of ensemble irradiance forecasts at seven locations,issued by the European Centre for Medium-Range Weather Forecasts,are calibrated via NCQRNN,as well as via an eclectic mix of benchmarking models,ranging from the naïve climatology to the state-of-the-art deep-learning and other non-crossing models.Formal and stringent forecast verification suggests that the forecasts post-processed via NCQRNN attain the maximum sharpness subject to calibration,amongst all competitors.Furthermore,the proposed conception to resolve quantile crossing is remarkably simple yet general,and thus has broad applicability as it can be integrated with many shallow-and deep-learning-based neural networks.

Intrusion Detection System for Smart Industrial Environments with Ensemble Feature Selection and Deep Convolutional Neural Networks

Smart Industrial environments use the Industrial Internet of Things(IIoT)for their routine operations and transform their industrial operations with intelligent and driven approaches.However,IIoT devices are vulnerable to cyber threats and exploits due to their connectivity with the internet.Traditional signature-based IDS are effective in detecting known attacks,but they are unable to detect unknown emerging attacks.Therefore,there is the need for an IDS which can learn from data and detect new threats.Ensemble Machine Learning(ML)and individual Deep Learning(DL)based IDS have been developed,and these individual models achieved low accuracy;however,their performance can be improved with the ensemble stacking technique.In this paper,we have proposed a Deep Stacked Neural Network(DSNN)based IDS,which consists of two stacked Convolutional Neural Network(CNN)models as base learners and Extreme Gradient Boosting(XGB)as the meta learner.The proposed DSNN model was trained and evaluated with the next-generation dataset,TON_IoT.Several pre-processing techniques were applied to prepare a dataset for the model,including ensemble feature selection and the SMOTE technique.Accuracy,precision,recall,F1-score,and false positive rates were used to evaluate the performance of the proposed ensemble model.Our experimental results showed that the accuracy for binary classification is 99.61%,which is better than in the baseline individual DL and ML models.In addition,the model proposed for IDS has been compared with similar models.The proposed DSNN achieved better performance metrics than the other models.The proposed DSNN model will be used to develop enhanced IDS for threat mitigation in smart industrial environments.

Neural network study of the nuclear ground-state spin distribution within a random interaction ensemble

The distribution of the nuclear ground-state spin in a two-body random ensemble(TBRE)was studied using a general classification neural network(NN)model with two-body interaction matrix elements as input features and the corresponding ground-state spins as labels or output predictions.The quantum many-body system problem exceeds the capability of our optimized NNs in terms of accurately predicting the ground-state spin of each sample within the TBRE.However,our NN model effectively captured the statistical properties of the ground-state spin because it learned the empirical regularity of the ground-state spin distribution in TBRE,as discovered by physicists.

Rapid assessment of flood loss based on neural network ensemble

Considering the defects of low accuracy and slow speed existing in traditional flood loss assessment, firstly, the technical route of flood loss assessment was presented based on the neural network ensemble. Secondly, through the study of certain country of Poyang Lake district, the flood loss assessment indicators of the test area were analyzed and extracted by utilizing analytic hierarchy process (AHP), and the weights of the impact factors were assigned. Subsequently, the approaches to generate individuals and conclusions of neural network ensemble model were also investigated. In the platform of C# language and neural network library under AForge.NET open source, a flood loss assessment program which could rapidly build neural network ensemble models was developed. Finally, the proposed method was tested and verified. The comparison results between the assessment results of the proposed method and the actual statistical flood loss proved the feasibility of this method, thus a new approach for flood loss assessment was provided.

Neural Network Ensemble Residual Kriging Application for Spatial Variability of Soil Properties

High quality, agricultural nutrient distribution maps are necessary for precision management, but depend on initial soil sample analyses and interpolation techniques. To examine the methodologies for and explore the capability of interpolating soil properties based on neural network ensemble residual kriging, a silage field at Hayes, Northern Ireland, UK, was selected for this study with all samples being split into independent training and validation data sets. The training data set, comprised of five soil properties: soil pH, soil available P, soil available K, soil available Mg and soil available S,was modeled for spatial variability using 1) neural network ensemble residual kriging, 2) neural network ensemble and 3) kriging with their accuracies being estimated by means of the validation data sets. Ordinary kriging of the residuals provided accurate local estimates, while final estimates were produced as a sum of the artificial neural network (ANN)ensemble estimates and the ordinary kriging estimates of the residuals. Compared to kriging and neural network ensemble,the neural network ensemble residual kriging achieved better or similar accuracy for predicting and estimating contour maps. Thus, the results demonstrated that ANN ensemble residual kriging was an efficient alternative to the conventional geo-statistical models that were usually used for interpolation of a data set in the soil science area.

Improving Land Resource Evaluation Using Fuzzy Neural Network Ensembles

Land evaluation factors often contain continuous-, discrete- and nominal-valued attributes. In traditional land evaluation, these different attributes are usually graded into categorical indexes by land resource experts, and the evaluation results rely heavily on experts＇ experiences. In order to overcome the shortcoming, we presented a fuzzy neural network ensemble method that did not require grading the evaluation factors into categorical indexes and could evaluate land resources by using the three kinds of attribute values directly. A fuzzy back propagation neural network （BPNN）, a fuzzy radial basis function neural network （RBFNN）, a fuzzy BPNN ensemble, and a fuzzy RBFNN ensemble were used to evaluate the land resources in Guangdong Province. The evaluation results by using the fuzzy BPNN ensemble and the fuzzy RBFNN ensemble were much better than those by using the single fuzzy BPNN and the single fuzzy RBFNN, and the error rate of the single fuzzy RBFNN or fuzzy RBFNN ensemble was lower than that of the single fuzzy BPNN or fuzzy BPNN ensemble, respectively. By using the fuzzy neural network ensembles, the validity of land resource evaluation was improved and reliance on land evaluators＇ experiences was considerably reduced.

Clustering-based selective neural network ensemble

An effective ensemble should consist of a set of networks that are both accurate and diverse. We propose a novel clustering-based selective algorithm for constructing neural network ensemble, where clustering technology is used to classify trained networks according to similarity and optimally select the most accurate individual network from each cluster to make up the ensemble. Empirical studies on regression of four typical datasets showed that this approach yields significantly smaller en- semble achieving better performance than other traditional ones such as Bagging and Boosting. The bias variance decomposition of the predictive error shows that the success of the proposed approach may lie in its properly tuning the bias/variance trade-off to reduce the prediction error (the sum of bias2 and variance).

Prediction of malignancy selective neural networks degree in brain glioma using ensemble

A clustering algorithm based selective neural networks ensemble （CLUSEN） is proposed to predict the degree of malignancy in brain glioma. Since the degree prediction of malignancy is critical before brain surgery, many learning methods are used like rule induction algorithm, single neural networks, support vector machines, etc. Ensemble learning methods can improve the generalization of single learning machine, and are becoming popular in the machine learning and medical data processing communities. The procedure of CLUSEN can efficiently remove redundancy learning individuals and help improve the diversity of ensemble methods. CLUSEN is used to predict the degree of malignancy in brain glioma. Experimental results on a set of brain glioma data show that, compared to support vector machines, rule induction and single neural networks, the classification accuracy of CLUSEN is higher.

Ensemble Recurrent Neural Network-Based Residual Useful Life Prognostics of Aircraft Engines

Residual useful life(RUL)prediction is a key issue for improving efficiency of aircraft engines and reducing their maintenance cost.Owing to various failure mechanism and operating environment,the application of classical models in RUL prediction of aircraft engines is fairly difficult.In this study,a novel RUL prognostics method based on using ensemble recurrent neural network to process massive sensor data is proposed.First of all,sensor data obtained from the aircraft engines are preprocessed to eliminate singular values,reduce random fluctuation and preserve degradation trend of the raw sensor data.Secondly,three kinds of recurrent neural networks(RNN),including ordinary RNN,long shortterm memory(LSTM),and gated recurrent unit(GRU),are individually constructed.Thirdly,ensemble learning mechanism is designed to merge the above RNNs for producing a more accurate RUL prediction.The effectiveness of the proposed method is validated using two characteristically different turbofan engine datasets.Experimental results show a competitive performance of the proposed method in comparison with typical methods reported in literatures.

Enhanced Detection of Glaucoma on Ensemble Convolutional Neural Network for Clinical Informatics

Irretrievable loss of vision is the predominant result of Glaucoma in the retina.Recently,multiple approaches have paid attention to the automatic detection of glaucoma on fundus images.Due to the interlace of blood vessels and the herculean task involved in glaucoma detection,the exactly affected site of the optic disc of whether small or big size cup,is deemed challenging.Spatially Based Ellipse Fitting Curve Model(SBEFCM)classification is suggested based on the Ensemble for a reliable diagnosis of Glaucomain theOptic Cup(OC)and Optic Disc(OD)boundary correspondingly.This research deploys the Ensemble Convolutional Neural Network(CNN)classification for classifying Glaucoma or Diabetes Retinopathy(DR).The detection of the boundary between the OC and the OD is performed by the SBEFCM,which is the latest weighted ellipse fitting model.The SBEFCM that enhances and widens the multi-ellipse fitting technique is proposed here.There is a preprocessing of input fundus image besides segmentation of blood vessels to avoid interlacing surrounding tissues and blood vessels.The ascertaining of OCandODboundary,which characterizedmany output factors for glaucoma detection,has been developed by EnsembleCNNclassification,which includes detecting sensitivity,specificity,precision,andArea Under the receiver operating characteristic Curve(AUC)values accurately by an innovative SBEFCM.In terms of contrast,the proposed Ensemble CNNsignificantly outperformed the current methods.

Neural Network Compact Ensemble and Its Applications

There has been many methods in constructing neural network （NN） ensembles, where the method of simultaneous training has succeed in generalization performance and efficiency. But just like regular methods of constructing NN ensembles, it follows the two steps, first training component networks, and then combining them. As the two steps being independent, an assumption is used to facilitate interactions among NNs during the training stage. This paper presents a compact ensemble method which integrates the two steps of ensemble construction into one step by attempting to train individual NNs in an ensemble and weigh the individual members adaptively according to their individual performance in the same learning process. This provides an opportunity for the individual NNs to interact with each other based on their real contributions to the ensemble. The classification performance of NN compact ensemble （NNCE） was validated through some benchmark problems in machine learning, including Australian credit card assessment, pima Indians diabetes, heart disease, breast cancer and glass. Compared with other ensembles, the classification error rate of NNCE can be decreased by 0.45% to 68%. In addition, the NNCE was applied to fault diagnosis for rolling element bearing. The 11 time-domain statistical features are extracted as the properties of data, and the NNCE is employed to classify the data. With the results of several experiments, the compact ensemble method is shown to give good generalization performance. The compact ensemble method can recognize the different fault types and various fault degrees of the same fault type.

From Designing A Single Neural Network to Designing Neural Network Ensembles

This paper introduces supervised learning model, and surveys related research work. The paper is organised as follows. A supervised learning model is firstly described. The bias-variance trade-off is then discussed for the supervised learning model. Based on the bias-variance trade-off, both the single neural network approaches and the neural network ensemble approaches are overviewed, and problems with the existing approaches are indicated. Finally, the paper concludes with specifying potential future research directions.

A New Selective Neural Network Ensemble Method Based on Error Vectorization and Its Application in High-density Polyethylene (HDPE) Cascade Reaction Process

Chemical processes are complex, for which traditional neural network models usually can not lead to satisfactory accuracy. Selective neural network ensemble is an effective way to enhance the generalization accuracy of networks, but there are some problems, e.g., lacking of unified definition of diversity among component neural networks and difficult to improve the accuracy by selecting if the diversities of available networks are small. In this study, the output errors of networks are vectorized, the diversity of networks is defined based on the error vectors, and the size of ensemble is analyzed. Then an error vectorization based selective neural network ensemble (EVSNE) is proposed, in which the error vector of each network can offset that of the other networks by training the component networks orderly. Thus the component networks have large diversity. Experiments and comparisons over standard data sets and actual chemical process data set for production of high-density polyethylene demonstrate that EVSNE performs better in generalization ability.

Shallow water bathymetry based on a back propagation neural network and ensemble learning using multispectral satellite imagery

The back propagation(BP)neural network method is widely used in bathymetry based on multispectral satellite imagery.However,the classical BP neural network method faces a potential problem because it easily falls into a local minimum,leading to model training failure.This study confirmed that the local minimum problem of the BP neural network method exists in the bathymetry field and cannot be ignored.Furthermore,to solve the local minimum problem of the BP neural network method,a bathymetry method based on a BP neural network and ensemble learning(BPEL)is proposed.First,the remote sensing imagery and training sample were used as input datasets,and the BP method was used as the base learner to produce multiple water depth inversion results.Then,a new ensemble strategy,namely the minimum outlying degree method,was proposed and used to integrate the water depth inversion results.Finally,an ensemble bathymetric map was acquired.Anda Reef,northeastern Jiuzhang Atoll,and Pingtan coastal zone were selected as test cases to validate the proposed method.Compared with the BP neural network method,the root-mean-square error and the average relative error of the BPEL method can reduce by 0.65–2.84 m and 16%–46%in the three test cases at most.The results showed that the proposed BPEL method could solve the local minimum problem of the BP neural network method and obtain highly robust and accurate bathymetric maps.

Structural Damage Identification Using Ensemble Deep Convolutional Neural Network Models

The existing strategy for evaluating the damage condition of structures mostly focuses on feedback supplied by traditional visualmethods,which may result in an unreliable damage characterization due to inspector subjectivity or insufficient level of expertise.As a result,a robust,reliable,and repeatable method of damage identification is required.Ensemble learning algorithms for identifying structural damage are evaluated in this article,which use deep convolutional neural networks,including simple averaging,integrated stacking,separate stacking,and hybridweighted averaging ensemble and differential evolution(WAE-DE)ensemblemodels.Damage identification is carried out on three types of damage.The proposed algorithms are used to analyze the damage of 4585 structural images.The effectiveness of the ensemble learning techniques is evaluated using the confusion matrix.For the testing dataset,the confusion matrix achieved an accuracy of 94 percent and a minimum recall of 92 percent for the best model(WAE-DE)in distinguishing damage types as flexural,shear,combined,or undamaged.

ENSOCOM: Ensemble of Multi-Output Neural Network’s Components for Multi-Label Classification

Multitasking and multioutput neural networks models jointly learn related classification tasks from a shared structure.Hard parameters sharing is a multitasking approach that shares hidden layers between multiple taskspecific outputs.The output layers’weights are essential in transforming aggregated neurons outputs into tasks labels.This paper redirects the multioutput network research to prove that the ensemble of output layers prediction can improve network performance in classifying multi-label classification tasks.The network’s output layers initialized with different weights simulate multiple semi-independent classifiers that can make non-identical label sets predictions for the same instance.The ensemble of a multi-output neural network that learns to classify the same multi-label classification task per output layer can outperform an individual output layer neural network.We propose an ensemble strategy of output layers components in the multi-output neural network for multi-label classification(ENSOCOM).The baseline and proposed models are selected based on the size of the hidden layer and the number of output layers to evaluate the proposed method comprehensively.The ENSOCOM method improved the performance of the neural networks on five different multi-label datasets based on several evaluation metrics.The methods presented in this work can substitute the standard labels representation and predictions generation of any neural network.

An Optimized Convolutional Neural Network Architecture Based on Evolutionary Ensemble Learning

Convolutional Neural Networks(CNNs)models succeed in vast domains.CNNs are available in a variety of topologies and sizes.The challenge in this area is to develop the optimal CNN architecture for a particular issue in order to achieve high results by using minimal computational resources to train the architecture.Our proposed framework to automated design is aimed at resolving this problem.The proposed framework is focused on a genetic algorithm that develops a population of CNN models in order to find the architecture that is the best fit.In comparison to the co-authored work,our proposed framework is concerned with creating lightweight architectures with a limited number of parameters while retaining a high degree of validity accuracy utilizing an ensemble learning technique.This architecture is intended to operate on low-resource machines,rendering it ideal for implementation in a number of environments.Four common benchmark image datasets are used to test the proposed framework,and it is compared to peer competitors’work utilizing a range of parameters,including accuracy,the number of model parameters used,the number of GPUs used,and the number of GPU days needed to complete the method.Our experimental findings demonstrated a significant advantage in terms of GPU days,accuracy,and the number of parameters in the discovered model.

Coronavirus Detection Using Two Step-AS Clustering and Ensemble Neural Network Model

This study presents a model of computer-aided intelligence capable of automatically detecting positive COVID-19 instances for use in regular medical applications.The proposed model is based on an Ensemble boosting Neural Network architecture and can automatically detect discriminatory features on chestX-ray images through Two Step-As clustering algorithm with rich filter families,abstraction and weight-sharing properties.In contrast to the generally used transformational learning approach,the proposed model was trained before and after clustering.The compilation procedure divides the datasets samples and categories into numerous sub-samples and subcategories and then assigns new group labels to each new group,with each subject group displayed as a distinct category.The retrieved characteristics discriminant cases were used to feed the Multiple Neural Network method,which was then utilised to classify the instances.The Two Step-AS clustering method has been modified by pre-aggregating the dataset before applying Multiple Neural Network algorithm to detect COVID-19 cases from chest X-ray findings.Models forMultiple Neural Network and Two Step-As clustering algorithms were optimised by utilising Ensemble Bootstrap Aggregating algorithm to reduce the number of hyper parameters they include.The testswere carried out using theCOVID-19 public radiology database,and a cross-validationmethod ensured accuracy.The proposed classifier with an accuracy of 98.02%percent was found to provide the most efficient outcomes possible.The result is a lowcost,quick and reliable intelligence tool for detecting COVID-19 infection.

Tissue specific prediction of N^(6)-methyladenine sites based on an ensemble of multi-input hybrid neural network

N^(6)-Methyladenine is a dynamic and reversible post translational modification,which plays an essential role in various biological processes.Because of the current inability to identify m6A-containing mRNAs,computational approaches have been developed to identify m6A sites in DNA sequences.Aiming to improve prediction performance,we introduced a novel ensemble computational approach based on three hybrid deep neural networks,including a convolutional neural network,a capsule network,and a bidirectional gated recurrent unit(BiGRU)with the self-attention mechanism,to identify m6A sites in four tissues of three species.Across a total of 11 datasets,we selected different feature subsets,after optimized from 4933 dimensional features,as input for the deep hybrid neural networks.In addition,to solve the deviation caused by the relatively small number of experimentally verified samples,we constructed an ensemble model through integrating five sub-classifiers based on different training datasets.When compared through 5-fold cross-validation and independent tests,our model showed its superiority to previous methods,im6A-TS-CNN and iRNA-m6A.

Ensemble Neural Network in Classifying Handwritten Arabic Numerals

A method has been proposed to classify handwritten Arabic numerals in its compressed form using partitioning approach, Leader algorithm and Neural network. Handwritten numerals are represented in a matrix form. Compressing the matrix representation by merging adjacent pair of rows using logical OR operation reduces its size in half. Considering each row as a partitioned portion, clusters are formed for same partition of same digit separately. Leaders of clusters of partitions are used to recognize the patterns by Divide and Conquer approach using proposed ensemble neural network. Experimental results show that the proposed method recognize the patterns accurately.