From Designing A Single Neural Network to Designing Neural Network Ensembles

This paper introduces supervised learning model, and surveys related research work. The paper is organised as follows. A supervised learning model is firstly described. The bias-variance trade-off is then discussed for the supervised learning model. Based on the bias-variance trade-off, both the single neural network approaches and the neural network ensemble approaches are overviewed, and problems with the existing approaches are indicated. Finally, the paper concludes with specifying potential future research directions.

Lightweight Network Ensemble Architecture for Environmental Perception on the Autonomous System

It is important for the autonomous system to understand environmental information.For the autonomous system,it is desirable to have a strong generalization ability to deal with different complex environmental information,as well as have high accuracy and quick inference speed.Network ensemble architecture is a good choice to improve network performance.However,it is unsuitable for real-time applications on the autonomous system.To tackle this problem,a new neural network ensemble named partial-shared ensemble network(PSENet)is presented.PSENet changes network ensemble architecture from parallel architecture to scatter architecture and merges multiple component networks together to accelerate the inference speed.To make component networks independent of each other,a training method is designed to train the network ensemble architecture.Experiments on Camvid and CIFAR-10 reveal that PSENet achieves quick inference speed while maintaining the ability of ensemble learning.In the real world,PSENet is deployed on the unmanned system and deals with vision tasks such as semantic segmentation and environmental prediction in different fields.

Deep neural network ensembles for remote sensing land cover and land use classification

With the advancement of satellite technology,a considerable amount of very high-resolution imagery has become available to be used for the Land Cover and Land Use(LCLU)classification task aiming to categorize remotely sensed images based on their semantic content.Recently,Deep Neural Networks(DNNs)have been widely used for different applications in the field of remote sensing and they have profound impacts;however,improvement of the generalizability and robustness of the DNNs needs to be progressed further to achieve higher accuracy for a variety of sensing geometries and categories.We address this problem by deploying three different Deep Neural Network Ensemble(DNNE)methods and creating a comparative analysis for the LCLU classification task.DNNE enables improvement of the performance of DNNs by ensuring the diversity of the models that are combined.Thus,enhances the generalizability of the models and produces more robust and generalizable outcomes for LCLU classification tasks.The experimental results on NWPU-RESISC45 and AID datasets demonstrate that utilizing the aggregated information from multiple DNNs leads to an increase in classification performance,achieves state-of-the-art,and promotes researchers to make use of DNNE.

Enhanced Detection of Glaucoma on Ensemble Convolutional Neural Network for Clinical Informatics

Irretrievable loss of vision is the predominant result of Glaucoma in the retina.Recently,multiple approaches have paid attention to the automatic detection of glaucoma on fundus images.Due to the interlace of blood vessels and the herculean task involved in glaucoma detection,the exactly affected site of the optic disc of whether small or big size cup,is deemed challenging.Spatially Based Ellipse Fitting Curve Model(SBEFCM)classification is suggested based on the Ensemble for a reliable diagnosis of Glaucomain theOptic Cup(OC)and Optic Disc(OD)boundary correspondingly.This research deploys the Ensemble Convolutional Neural Network(CNN)classification for classifying Glaucoma or Diabetes Retinopathy(DR).The detection of the boundary between the OC and the OD is performed by the SBEFCM,which is the latest weighted ellipse fitting model.The SBEFCM that enhances and widens the multi-ellipse fitting technique is proposed here.There is a preprocessing of input fundus image besides segmentation of blood vessels to avoid interlacing surrounding tissues and blood vessels.The ascertaining of OCandODboundary,which characterizedmany output factors for glaucoma detection,has been developed by EnsembleCNNclassification,which includes detecting sensitivity,specificity,precision,andArea Under the receiver operating characteristic Curve(AUC)values accurately by an innovative SBEFCM.In terms of contrast,the proposed Ensemble CNNsignificantly outperformed the current methods.

Neural Network Compact Ensemble and Its Applications

There has been many methods in constructing neural network （NN） ensembles, where the method of simultaneous training has succeed in generalization performance and efficiency. But just like regular methods of constructing NN ensembles, it follows the two steps, first training component networks, and then combining them. As the two steps being independent, an assumption is used to facilitate interactions among NNs during the training stage. This paper presents a compact ensemble method which integrates the two steps of ensemble construction into one step by attempting to train individual NNs in an ensemble and weigh the individual members adaptively according to their individual performance in the same learning process. This provides an opportunity for the individual NNs to interact with each other based on their real contributions to the ensemble. The classification performance of NN compact ensemble （NNCE） was validated through some benchmark problems in machine learning, including Australian credit card assessment, pima Indians diabetes, heart disease, breast cancer and glass. Compared with other ensembles, the classification error rate of NNCE can be decreased by 0.45% to 68%. In addition, the NNCE was applied to fault diagnosis for rolling element bearing. The 11 time-domain statistical features are extracted as the properties of data, and the NNCE is employed to classify the data. With the results of several experiments, the compact ensemble method is shown to give good generalization performance. The compact ensemble method can recognize the different fault types and various fault degrees of the same fault type.

A new ensemble feature selection and its application to pattern classification

Neural network ensemble based on rough sets reduct is proposed to decrease the computational complexity of conventional ensemble feature selection algorithm. First, a dynamic reduction technology combining genetic algorithm with resampling method is adopted to obtain reducts with good generalization ability. Second, Multiple BP neural networks based on different reducts are built as base classifiers. According to the idea of selective ensemble, the neural network ensemble with best generalization ability can be found by search strategies. Finally, classification based on neural network ensemble is implemented by combining the predictions of component networks with voting. The method has been verified in the experiment of remote sensing image and five UCI datasets classification. Compared with conventional ensemble feature selection algorithms, it costs less time and lower computing complexity, and the classification accuracy is satisfactory.

Ensemble Prediction of Monsoon Index with a Genetic Neural Network Model

After the consideration of the nonlinear nature changes of monsoon index,and the subjective determination of network structure in traditional artificial neural network prediction modeling,monthly and seasonal monsoon intensity index prediction is studied in this paper by using nonlinear genetic neural network ensemble prediction（GNNEP）modeling.It differs from traditional prediction modeling in the following aspects： （1）Input factors of the GNNEP model of monsoon index were selected from a large quantity of preceding period high correlation factors,such as monthly sea temperature fields,monthly 500-hPa air temperature fields,monthly 200-hPa geopotential height fields,etc.,and they were also highly information-condensed and system dimensionality-reduced by using the empirical orthogonal function（EOF）method,which effectively condensed the useful information of predictors and therefore controlled the size of network structure of the GNNEP model.（2）In the input design of the GNNEP model,a mean generating function（MGF）series of predictand（monsoon index）was added as an input factor;the contrast analysis of results of predic- tion experiments by a physical variable predictor-predictand MGF GNNEP model and a physical variable predictor GNNEP model shows that the incorporation of the periodical variation of predictand（monsoon index）is very effective in improving the prediction of monsoon index.（3）Different from the traditional neural network modeling,the GNNEP modeling is able to objectively determine the network structure of the GNNNEP model,and the model constructed has a better generalization capability.In the case of identical predictors,prediction modeling samples,and independent prediction samples,the prediction accuracy of our GNNEP model combined with the system dimensionality reduction technique of predictors is clearly higher than that of the traditional stepwise regression model using the traditional treatment technique of predictors,suggesting that the GNNEP model opens up a vast range of possibilities for operational weather prediction.

A Cause-Selecting Control Chart Method for Monitoring and Diagnosing Dependent Manufacturing Process Stages

Many industrial products are normally processed through multiple manufacturing process stages before it becomes a final product.Statistical process control techniques often utilize standard Shewhart control charts to monitor these process stages.If the process stages are independent,this is a meaningful procedure.However,they are not independent in many manufacturing scenarios.The standard Shewhart control charts can not provide the information to determine which process stage or group of process stages has caused the problems(i.e.,standard Shewhart control charts could not diagnose dependent manufacturing process stages).This study proposes a selective neural network ensemble-based cause-selecting system of control charts to monitor these process stages and distinguish incoming quality problems and problems in the current stage of a manufacturing process.Numerical results show that the proposed method is an improvement over the use of separate Shewhart control chart for each of dependent process stages,and even ordinary quality practitioners who lack of expertise in theoretical analysis can implement regression estimation and neural computing readily.

Analysis of Negative Correlation Learning

This paper describes negative correlation learning for designing neural network ensembles. Negative correlation learning has been firstly analysed in terms of minimising mutual information on a regression task. By minimising the mutual information between variables extracted by two neural networks, they are forced to convey different information a-bout some features of their input. Based on the decision boundaries and correct response sets, negative correlation learning has been further studied on two pattern classification problems. The purpose of examining the decision boundaries and the correct response sets is not only to illustrate the learning behavior of negative correlation learning, but also to cast light on how to design more effective neural network ensembles. The experimental results showed the decision boundary of the trained neural network ensemble by negative correlation learning is almost as good as the optimum decision boundary.

Recent Advances in Evolutionary Computation

Evolutionary computation has experienced a tremendous growth in the last decade in both theoretical analyses and industrial applications. Its scope has evolved beyond its original meaning of ＂biological evolution＂ toward a wide variety of nature inspired computational algorithms and techniques, including evolutionary, neural, ecological, social and economical computation, etc, in a unified framework. Many research topics in evolutionary computation nowadays are not necessarily ＂evolutionary＂. This paper provides an overview of some recent advances in evolutionary computation that have been made in CERCIA at the University of Birmingham, UK. It covers a wide range of topics in optimization, learning and design using evolutionary approaches and techniques, and theoretical results in the computational time complexity of evolutionary algorithms. Some issues related to future development of evolutionary computation are also discussed.