Low rank optimization for efficient deep learning:making a balance between compact architecture and fast training

Deep neural networks(DNNs)have achieved great success in many data processing applications.However,high computational complexity and storage cost make deep learning difficult to be used on resource-constrained devices,and it is not environmental-friendly with much power cost.In this paper,we focus on low-rank optimization for efficient deep learning techniques.In the space domain,DNNs are compressed by low rank approximation of the network parameters,which directly reduces the storage requirement with a smaller number of network parameters.In the time domain,the network parameters can be trained in a few subspaces,which enables efficient training for fast convergence.The model compression in the spatial domain is summarized into three categories as pre-train,pre-set,and compression-aware methods,respectively.With a series of integrable techniques discussed,such as sparse pruning,quantization,and entropy coding,we can ensemble them in an integration framework with lower computational complexity and storage.In addition to summary of recent technical advances,we have two findings for motivating future works.One is that the effective rank,derived from the Shannon entropy of the normalized singular values,outperforms other conventional sparse measures such as the?_1 norm for network compression.The other is a spatial and temporal balance for tensorized neural networks.For accelerating the training of tensorized neural networks,it is crucial to leverage redundancy for both model compression and subspace training.

Deep learning algorithm featuring continuous learning for modulation classifications in wireless networks

Although modulation classification based on deep neural network can achieve high Modulation Classification(MC)accuracies,catastrophic forgetting will occur when the neural network model continues to learn new tasks.In this paper,we simulate the dynamic wireless communication environment and focus on breaking the learning paradigm of isolated automatic MC.We innovate a research algorithm for continuous automatic MC.Firstly,a memory for storing representative old task modulation signals is built,which is employed to limit the gradient update direction of new tasks in the continuous learning stage to ensure that the loss of old tasks is also in a downward trend.Secondly,in order to better simulate the dynamic wireless communication environment,we employ the mini-batch gradient algorithm which is more suitable for continuous learning.Finally,the signal in the memory can be replayed to further strengthen the characteristics of the old task signal in the model.Simulation results verify the effectiveness of the method.

Recent Progresses in Deep Learning Based Acoustic Models

In this paper,we summarize recent progresses made in deep learning based acoustic models and the motivation and insights behind the surveyed techniques.We first discuss models such as recurrent neural networks(RNNs) and convolutional neural networks(CNNs) that can effectively exploit variablelength contextual information,and their various combination with other models.We then describe models that are optimized end-to-end and emphasize on feature representations learned jointly with the rest of the system,the connectionist temporal classification(CTC) criterion,and the attention-based sequenceto-sequence translation model.We further illustrate robustness issues in speech recognition systems,and discuss acoustic model adaptation,speech enhancement and separation,and robust training strategies.We also cover modeling techniques that lead to more efficient decoding and discuss possible future directions in acoustic model research.

Optimizing Deep Learning Parameters Using Genetic Algorithm for Object Recognition and Robot Grasping

The performance of deep learning(DL)networks has been increased by elaborating the network structures. However, the DL netowrks have many parameters, which have a lot of influence on the performance of the network. We propose a genetic algorithm(GA) based deep belief neural network(DBNN) method for robot object recognition and grasping purpose. This method optimizes the parameters of the DBNN method, such as the number of hidden units, the number of epochs, and the learning rates, which would reduce the error rate and the network training time of object recognition. After recognizing objects, the robot performs the pick-andplace operations. We build a database of six objects for experimental purpose. Experimental results demonstrate that our method outperforms on the optimized robot object recognition and grasping tasks.

Deep learning for fast channel estimation in millimeter-wave MIMO systems

Channel estimation has been considered as a key issue in the millimeter-wave(mmWave)massive multi-input multioutput(MIMO)communication systems,which becomes more challenging with a large number of antennas.In this paper,we propose a deep learning(DL)-based fast channel estimation method for mmWave massive MIMO systems.The proposed method can directly and effectively estimate channel state information(CSI)from received data without performing pilot signals estimate in advance,which simplifies the estimation process.Specifically,we develop a convolutional neural network(CNN)-based channel estimation network for the case of dimensional mismatch of input and output data,subsequently denoted as channel(H)neural network(HNN).It can quickly estimate the channel information by learning the inherent characteristics of the received data and the relationship between the received data and the channel,while the dimension of the received data is much smaller than the channel matrix.Simulation results show that the proposed HNN can gain better channel estimation accuracy compared with existing schemes.

Deep learning-based fault diagnostic network of high-speed train secondary suspension systems for immunity to track irregularities and wheel wear

Fault detection and isolation of high-speed train suspension systems is of critical importance to guarantee train running safety. Firstly, the existing methods concerning fault detection or isolation of train suspension systems are briefly reviewed and divided into two categories, i.e., model-based and data-driven approaches. The advantages and disadvantages of these two categories of approaches are briefly summarized. Secondly, a 1D convolution network-based fault diagnostic method for highspeed train suspension systems is designed. To improve the robustness of the method, a Gaussian white noise strategy(GWN-strategy) for immunity to track irregularities and an edge sample training strategy(EST-strategy) for immunity to wheel wear are proposed. The whole network is called GWN-EST-1 DCNN method. Thirdly, to show the performance of this method, a multibody dynamics simulation model of a high-speed train is built to generate the lateral acceleration of a bogie frame corresponding to different track irregularities, wheel profiles, and secondary suspension faults. The simulated signals are then inputted into the diagnostic network, and the results show the correctness and superiority of the GWN-EST-1DCNN method. Finally,the 1DCNN method is further validated using tracking data of a CRH3 train running on a high-speed railway line.

A Hierarchy Distributed-Agents Model for Network Risk Evaluation Based on Deep Learning

Deep Learning presents a critical capability to be geared into environments being constantly changed and ongoing learning dynamic,which is especially relevant in Network Intrusion Detection.In this paper,as enlightened by the theory of Deep Learning Neural Networks,Hierarchy Distributed-Agents Model for Network Risk Evaluation,a newly developed model,is proposed.The architecture taken on by the distributed-agents model are given,as well as the approach of analyzing network intrusion detection using Deep Learning,the mechanism of sharing hyper-parameters to improve the efficiency of learning is presented,and the hierarchical evaluative framework for Network Risk Evaluation of the proposed model is built.Furthermore,to examine the proposed model,a series of experiments were conducted in terms of NSLKDD datasets.The proposed model was able to differentiate between normal and abnormal network activities with an accuracy of 97.60%on NSL-KDD datasets.As the results acquired from the experiment indicate,the model developed in this paper is characterized by high-speed and high-accuracy processing which shall offer a preferable solution with regard to the Risk Evaluation in Network.

Identifi cation of paralytic shellfi sh toxin-producing microalgae using machine learning and deep learning methods

Paralytic shellfi sh poisoning(PSP)microalgae,as one of the harmful algal blooms,causes great damage to the of fshore fi shery,marine culture,and marine ecological environment.At present,there is no technique for real-time accurate identifi cation of toxic microalgae,by combining three-dimensional fluorescence with machine learning(ML)and deep learning(DL),we developed methods to classify the PSP and non-PSP microalgae.The average classifi cation accuracies of these two methods for microalgae are above 90%,and the accuracies for discriminating 12 microalgae species in PSP and non-PSP microalgae are above 94%.When the emission wavelength is 650-690 nm,the fl uorescence characteristics bands(excitation wavelength)occur dif ferently at 410-480 nm and 500-560 nm for PSP and non-PSP microalgae,respectively.The identification accuracies of ML models(support vector machine(SVM),and k-nearest neighbor rule(k-NN)),and DL model(convolutional neural network(CNN))to PSP microalgae are 96.25%,96.36%,and 95.88%respectively,indicating that ML and DL are suitable for the classifi cation of toxic microalgae.

Modified Anam-Net Based Lightweight Deep Learning Model for Retinal Vessel Segmentation

The accurate segmentation of retinal vessels is a challenging taskdue to the presence of various pathologies as well as the low-contrast ofthin vessels and non-uniform illumination. In recent years, encoder-decodernetworks have achieved outstanding performance in retinal vessel segmentation at the cost of high computational complexity. To address the aforementioned challenges and to reduce the computational complexity, we proposea lightweight convolutional neural network (CNN)-based encoder-decoderdeep learning model for accurate retinal vessels segmentation. The proposeddeep learning model consists of encoder-decoder architecture along withbottleneck layers that consist of depth-wise squeezing, followed by fullconvolution, and finally depth-wise stretching. The inspiration for the proposed model is taken from the recently developed Anam-Net model, whichwas tested on CT images for COVID-19 identification. For our lightweightmodel, we used a stack of two 3 × 3 convolution layers (without spatialpooling in between) instead of a single 3 × 3 convolution layer as proposedin Anam-Net to increase the receptive field and to reduce the trainableparameters. The proposed method includes fewer filters in all convolutionallayers than the original Anam-Net and does not have an increasing numberof filters for decreasing resolution. These modifications do not compromiseon the segmentation accuracy, but they do make the architecture significantlylighter in terms of the number of trainable parameters and computation time.The proposed architecture has comparatively fewer parameters (1.01M) thanAnam-Net (4.47M), U-Net (31.05M), SegNet (29.50M), and most of the otherrecent works. The proposed model does not require any problem-specificpre- or post-processing, nor does it rely on handcrafted features. In addition,the attribute of being efficient in terms of segmentation accuracy as well aslightweight makes the proposed method a suitable candidate to be used in thescreening platforms at the point of care. We evaluated our proposed modelon open-access datasets namely, DRIVE, STARE, and CHASE_DB. Theexperimental results show that the proposed model outperforms several stateof-the-art methods, such as U-Net and its variants, fully convolutional network (FCN), SegNet, CCNet, ResWNet, residual connection-based encoderdecoder network (RCED-Net), and scale-space approx. network (SSANet) in terms of {dice coefficient, sensitivity (SN), accuracy (ACC), and the areaunder the ROC curve (AUC)} with the scores of {0.8184, 0.8561, 0.9669, and0.9868} on the DRIVE dataset, the scores of {0.8233, 0.8581, 0.9726, and0.9901} on the STARE dataset, and the scores of {0.8138, 0.8604, 0.9752,and 0.9906} on the CHASE_DB dataset. Additionally, we perform crosstraining experiments on the DRIVE and STARE datasets. The result of thisexperiment indicates the generalization ability and robustness of the proposedmodel.

A Deep Learning-Based Continuous Blood Pressure Measurement by Dual Photoplethysmography Signals

This study proposed a measurement platform for continuous blood pressure estimation based on dual photoplethysmography(PPG)sensors and a deep learning(DL)that can be used for continuous and rapid measurement of blood pressure and analysis of cardiovascular-related indicators.The proposed platform measured the signal changes in PPG and converted them into physiological indicators,such as pulse transit time(PTT),pulse wave velocity(PWV),perfusion index(PI)and heart rate(HR);these indicators were then fed into the DL to calculate blood pressure.The hardware of the experiment comprised 2 PPG components(i.e.,Raspberry Pi 3 Model B and analog-todigital converter[MCP3008]),which were connected using a serial peripheral interface.The DL algorithm converted the stable dual PPG signals acquired from the strictly standardized experimental process into various physiological indicators as input parameters and finally obtained the systolic blood pressure(SBP),diastolic blood pressure(DBP)and mean arterial pressure(MAP).To increase the robustness of the DL model,this study input data of 100 Asian participants into the training database,including those with and without cardiovascular disease,each with a proportion of approximately 50%.The experimental results revealed that the mean absolute error and standard deviation of SBP was 0.17±0.46 mmHg.The mean absolute error and standard deviation of DBP was 0.27±0.52 mmHg.The mean absolute error and standard deviation of MAP was 0.16±0.40 mmHg.

Deep learning-based time-varying channel estimation with basis expansion model for MIMO-OFDM system

For high-speed mobile MIMO-OFDM system,a low-complexity deep learning(DL) based timevarying channel estimation scheme is proposed.To reduce the number of estimated parameters,the basis expansion model(BEM) is employed to model the time-varying channel,which converts the channel estimation into the estimation of the basis coefficient.Specifically,the initial basis coefficients are firstly used to train the neural network in an offline manner,and then the high-precision channel estimation can be obtained by small number of inputs.Moreover,the linear minimum mean square error(LMMSE) estimated channel is considered for the loss function in training phase,which makes the proposed method more practical.Simulation results show that the proposed method has a better performance and lower computational complexity compared with the available schemes,and it is robust to the fast time-varying channel in the high-speed mobile scenarios.

Tuning-up Learning Parameters for Deep Convolutional Neural Network:A Case Study for Hand-Drawn Sketch Images

Several recent successes in deep learning(DL),such as state-of-the-art performance on several image classification benchmarks,have been achieved through the improved configuration.Hyperparameters(HPs)tuning is a key factor affecting the performance of machine learning(ML)algorithms.Various state-of-the-art DL models use different HPs in different ways for classification tasks on different datasets.This manuscript provides a brief overview of learning parameters and configuration techniques to show the benefits of using a large-scale handdrawn sketch dataset for classification problems.We analyzed the impact of different learning parameters and toplayer configurations with batch normalization(BN)and dropouts on the performance of the pre-trained visual geometry group 19(VGG-19).The analyzed learning parameters include different learning rates and momentum values of two different optimizers,such as stochastic gradient descent(SGD)and Adam.Our analysis demonstrates that using the SGD optimizer and learning parameters,such as small learning rates with high values of momentum,along with both BN and dropouts in top layers,has a good impact on the sketch image classification accuracy.

Deep Learning ResNet101 Deep Features of Portable Chest X-Ray Accurately Classify COVID-19 Lung Infection

This study is designed to develop Artificial Intelligence(AI)based analysis tool that could accurately detect COVID-19 lung infections based on portable chest x-rays(CXRs).The frontline physicians and radiologists suffer from grand challenges for COVID-19 pandemic due to the suboptimal image quality and the large volume of CXRs.In this study,AI-based analysis tools were developed that can precisely classify COVID-19 lung infection.Publicly available datasets of COVID-19(N=1525),non-COVID-19 normal(N=1525),viral pneumonia(N=1342)and bacterial pneumonia(N=2521)from the Italian Society of Medical and Interventional Radiology(SIRM),Radiopaedia,The Cancer Imaging Archive(TCIA)and Kaggle repositories were taken.A multi-approach utilizing deep learning ResNet101 with and without hyperparameters optimization was employed.Additionally,the fea-tures extracted from the average pooling layer of ResNet101 were used as input to machine learning(ML)algorithms,which twice trained the learning algorithms.The ResNet101 with optimized parameters yielded improved performance to default parameters.The extracted features from ResNet101 are fed to the k-nearest neighbor(KNN)and support vector machine(SVM)yielded the highest 3-class classification performance of 99.86%and 99.46%,respectively.The results indicate that the proposed approach can be bet-ter utilized for improving the accuracy and diagnostic efficiency of CXRs.The proposed deep learning model has the potential to improve further the efficiency of the healthcare systems for proper diagnosis and prognosis of COVID-19 lung infection.

Improved Soil Quality Prediction Model Using Deep Learning for Smart Agriculture Systems

Soil is the major source of infinite lives on Earth and the quality of soil plays significant role on Agriculture practices all around.Hence,the evaluation of soil quality is very important for determining the amount of nutrients that the soil require for proper yield.In present decade,the application of deep learning models in many fields of research has created greater impact.The increasing soil data availability of soil data there is a greater demand for the remotely avail open source model,leads to the incorporation of deep learning method to predict the soil quality.With that concern,this paper proposes a novel model called Improved Soil Quality Prediction Model using Deep Learning(ISQP-DL).The work considers the chemical,physical and biological factors of soil in particular area to estimate the soil quality.Firstly,pH rating of soil samples has been collected from the soil testing laboratory from which the acidic range has been categorized through soil test and the same data has been taken as input to the Deep Neural Network Regression(DNNR)model.Secondly,soil nutrient data has been given as second input to the DNNR model.By utilizing this data set,the DNNR method is used to evaluate the fertility rate by which the soil quality has been estimated.For training and testing,the model uses Deep Neural Network Regression(DNNR),by utilizing the dataset.The results show that the proposed model is effective for SQP(Soil Quality Prediction Model)with efficient good fitting and generality is enhanced with input features with higher rate of classification accuracy.The results show that the proposed model achieves 96.7%of accuracy rate compared with existing models.

Cryptographic Based Secure Model on Dataset for Deep Learning Algorithms

Deep learning(DL)algorithms have been widely used in various security applications to enhance the performances of decision-based models.Malicious data added by an attacker can cause several security and privacy problems in the operation of DL models.The two most common active attacks are poisoning and evasion attacks,which can cause various problems,including wrong prediction and misclassification of decision-based models.Therefore,to design an efficient DL model,it is crucial to mitigate these attacks.In this regard,this study proposes a secure neural network(NN)model that provides data security during model training and testing phases.The main idea is to use cryptographic functions,such as hash function(SHA512)and homomorphic encryption(HE)scheme,to provide authenticity,integrity,and confidentiality of data.The performance of the proposed model is evaluated by experiments based on accuracy,precision,attack detection rate(ADR),and computational cost.The results show that the proposed model has achieved an accuracy of 98%,a precision of 0.97,and an ADR of 98%,even for a large number of attacks.Hence,the proposed model can be used to detect attacks and mitigate the attacker motives.The results also show that the computational cost of the proposed model does not increase with model complexity.

Prediction Model for Coronavirus Pandemic Using Deep Learning

The recent global outbreak of COVID-19 damaged the world health systems,human health,economy,and daily life badly.None of the countries was ready to face this emerging health challenge.Health professionals were not able to predict its rise and next move,as well as the future curve and impact on lives in case of a similar pandemic situation happened.This created huge chaos globally,for longer and the world is still struggling to come up with any suitable solution.Here the better use of advanced technologies,such as artificial intelligence and deep learning,may aid healthcare practitioners in making reliable COVID-19 diagnoses.The proposed research would provide a prediction model that would use Artificial Intelligence and Deep Learning to improve the diagnostic process by reducing unreliable diagnostic interpretation of chest CT scans and allowing clinicians to accurately discriminate between patients who are sick with COVID-19 or pneumonia,and also empowering health professionals to distinguish chest CT scans of healthy people.The efforts done by the Saudi government for the management and control of COVID-19 are remarkable,however;there is a need to improve the diagnostics process for better perception.We used a data set from Saudi regions to build a prediction model that can help distinguish between COVID-19 cases and regular cases from CT scans.The proposed methodology was compared to current models and found to be more accurate(93 percent)than the existing methods.

Person-Dependent Handwriting Verification for Special Education Using DeepLearning

Individuals with special needs learn more slowly than their peers and they need repetitions to be permanent.However,in crowded classrooms,it is dif-ficult for a teacher to deal with each student individually.This problem can be overcome by using supportive education applications.However,the majority of such applications are not designed for special education and therefore they are not efficient as expected.Special education students differ from their peers in terms of their development,characteristics,and educational qualifications.The handwriting skills of individuals with special needs are lower than their peers.This makes the task of Handwriting Recognition(HWR)more difficult.To over-come this problem,we propose a new personalized handwriting verification sys-tem that validates digits from the handwriting of special education students.The system uses a Convolutional Neural Network(CNN)created and trained from scratch.The data set used is obtained by collecting the handwriting of the students with the help of a tablet.A special education center is visited and the handwrittenfigures of the students are collected under the supervision of special education tea-chers.The system is designed as a person-dependent system as every student has their writing style.Overall,the system achieves promising results,reaching a recognition accuracy of about 94%.Overall,the system can verify special educa-tion students’handwriting digits with high accuracy and is ready to integrate with a mobile application that is designed to teach digits to special education students.

基于EasyDL平台的甘肃陇南核桃主要病害诊断模型的构建及应用

旨在解决甘肃陇南地区核桃产业中病虫害准确鉴定的问题,以提高种植户核桃园管理的水平和能力。基于自动化深度学习技术(Automated deep learning technology,AutoDL),利用机器学习模型搜索(Model Search)实现自动化人工智能(Automated artificial intelligence,AutoML)算法框架,基于飞桨开源深度学习平台Easy DL构建甘肃陇南核桃主要病害的诊断模型,并进行诊断精度的模型训练。结果显示,共246张训练集进入模型,鉴定为9种常见核桃病害,模型部署在API公用云上,通过微信小程序或浏览器运行。经训练,其诊断准确率达95%以上。说明通过EasyDL构建的陇南地区核桃上常见病害模型运行可靠,能够为核桃种植户提供准确的病害诊断,从而很好地指导种植户提高管理核桃园的水平和应付突发植保问题的能力,以便及时、迅速采取综合防治措施,最大程度地减少因病害造成的经济损失。同时也是相关从业人员和基层研究人员解决核桃植保问题的得力辅助工具。

Deep learning based Doppler frequency offset estimation for 5G-NR downlink in HSR scenario

In the fifth-generation new radio(5G-NR) high-speed railway(HSR) downlink,a deep learning(DL) based Doppler frequency offset(DFO) estimation scheme is proposed by using the back propagation neural network(BPNN).The proposed method mainly includes pre-training,training,and estimation phases,where the pre-training and training belong to the off-line stage,and the estimation is the online stage.To reduce the performance loss caused by the random initialization,the pre-training method is employed to acquire a desirable initialization,which is used as the initial parameters of the training phase.Moreover,the initial DFO estimation is used as input along with the received pilots to further improve the estimation accuracy.Different from the training phase,the initial DFO estimation in pre-training phase is obtained by the data and pilot symbols.Simulation results show that the mean squared error(MSE) performance of the proposed method is better than those of the available algorithms,and it has acceptable computational complexity.

Evaluation of a deep learning supported remote diagnosis model for identification of diabetic retinopathy using wide-field Optomap

Background:We test a deep learning(DL)supported remote diagnosis approach to detect diabetic retinopathy(DR)and other referable retinal pathologies using ultra-wide-field(UWF)Optomap.Methods:Prospective,non-randomized study involving diabetic patients seen at endocrinology clinics.Non-expert imagers were trained to obtain non-dilated images using UWF Primary.Images were graded by two retina specialists and classified as DR or incidental retinal findings.Cohen’s kappa was used to test the agreement between the remote diagnosis and the gold standard exam.A novel DL model was trained to identify the presence or absence of referable pathology,and sensitivity,specificity and area under the receiver operator characteristics curve(AUROC)were used to assess its performance.Results:A total of 265 patients were enrolled,of which 241 patients were imaged(433 eyes).The mean age was 50±17 years,45%of patients were female,34%had a diagnosis of diabetes mellitus type 1,and 66%of type 2.The average Hemoglobin A1c was 8.8±2.3%,and 81%were on Insulin.Of the 433 images,404(93%)were gradable,64 patients(27%)were referred to a retina specialist,and 46(19%)were referred to comprehensive ophthalmologist for a referable retinal pathology on remote diagnosis.Cohen’s kappa was 0.58,indicating moderate agreement.Our DL algorithm achieved an accuracy of 82.8%(95%CI:80.3-85.2%),a sensitivity of 81.0%(95%CI:78.5-83.6%),specificity of 73.5%(95%CI:70.6-76.3%),and AUROC of 81.0%(95%CI:78.5-83.6%).Conclusions:UWF Primary can be used in the non-ophthalmology setting to screen for referable retinal pathology and can be successfully supported by an automated algorithm for image classification.