L_(1)-Smooth SVM with Distributed Adaptive Proximal Stochastic Gradient Descent with Momentum for Fast Brain Tumor Detection

Brain tumors come in various types,each with distinct characteristics and treatment approaches,making manual detection a time-consuming and potentially ambiguous process.Brain tumor detection is a valuable tool for gaining a deeper understanding of tumors and improving treatment outcomes.Machine learning models have become key players in automating brain tumor detection.Gradient descent methods are the mainstream algorithms for solving machine learning models.In this paper,we propose a novel distributed proximal stochastic gradient descent approach to solve the L_(1)-Smooth Support Vector Machine(SVM)classifier for brain tumor detection.Firstly,the smooth hinge loss is introduced to be used as the loss function of SVM.It avoids the issue of nondifferentiability at the zero point encountered by the traditional hinge loss function during gradient descent optimization.Secondly,the L_(1) regularization method is employed to sparsify features and enhance the robustness of the model.Finally,adaptive proximal stochastic gradient descent(PGD)with momentum,and distributed adaptive PGDwithmomentum(DPGD)are proposed and applied to the L_(1)-Smooth SVM.Distributed computing is crucial in large-scale data analysis,with its value manifested in extending algorithms to distributed clusters,thus enabling more efficient processing ofmassive amounts of data.The DPGD algorithm leverages Spark,enabling full utilization of the computer’s multi-core resources.Due to its sparsity induced by L_(1) regularization on parameters,it exhibits significantly accelerated convergence speed.From the perspective of loss reduction,DPGD converges faster than PGD.The experimental results show that adaptive PGD withmomentumand its variants have achieved cutting-edge accuracy and efficiency in brain tumor detection.Frompre-trained models,both the PGD andDPGD outperform other models,boasting an accuracy of 95.21%.

Efficient and High-quality Recommendations via Momentum-incorporated Parallel Stochastic Gradient Descent-Based Learning

A recommender system(RS)relying on latent factor analysis usually adopts stochastic gradient descent(SGD)as its learning algorithm.However,owing to its serial mechanism,an SGD algorithm suffers from low efficiency and scalability when handling large-scale industrial problems.Aiming at addressing this issue,this study proposes a momentum-incorporated parallel stochastic gradient descent(MPSGD)algorithm,whose main idea is two-fold:a)implementing parallelization via a novel datasplitting strategy,and b)accelerating convergence rate by integrating momentum effects into its training process.With it,an MPSGD-based latent factor(MLF)model is achieved,which is capable of performing efficient and high-quality recommendations.Experimental results on four high-dimensional and sparse matrices generated by industrial RS indicate that owing to an MPSGD algorithm,an MLF model outperforms the existing state-of-the-art ones in both computational efficiency and scalability.

Convergence of Stochastic Gradient Descent in Deep Neural Network

Stochastic gradient descent(SGD) is one of the most common optimization algorithms used in pattern recognition and machine learning.This algorithm and its variants are the preferred algorithm while optimizing parameters of deep neural network for their advantages of low storage space requirement and fast computation speed.Previous studies on convergence of these algorithms were based on some traditional assumptions in optimization problems.However,the deep neural network has its unique properties.Some assumptions are inappropriate in the actual optimization process of this kind of model.In this paper,we modify the assumptions to make them more consistent with the actual optimization process of deep neural network.Based on new assumptions,we studied the convergence and convergence rate of SGD and its two common variant algorithms.In addition,we carried out numerical experiments with LeNet-5,a common network framework,on the data set MNIST to verify the rationality of our assumptions.

Stochastic Gradient Compression for Federated Learning over Wireless Network

As a mature distributed machine learning paradigm,federated learning enables wireless edge devices to collaboratively train a shared AI-model by stochastic gradient descent(SGD).However,devices need to upload high-dimensional stochastic gradients to edge server in training,which cause severe communication bottleneck.To address this problem,we compress the communication by sparsifying and quantizing the stochastic gradients of edge devices.We first derive a closed form of the communication compression in terms of sparsification and quantization factors.Then,the convergence rate of this communicationcompressed system is analyzed and several insights are obtained.Finally,we formulate and deal with the quantization resource allocation problem for the goal of minimizing the convergence upper bound,under the constraint of multiple-access channel capacity.Simulations show that the proposed scheme outperforms the benchmarks.

A Stochastic Gradient Descent Method for Computational Design of Random Rough Surfaces in Solar Cells

In this work,we develop a stochastic gradient descent method for the computational optimal design of random rough surfaces in thin-film solar cells.We formulate the design problems as random PDE-constrained optimization problems and seek the optimal statistical parameters for the random surfaces.The optimizations at fixed frequency as well as at multiple frequencies and multiple incident angles are investigated.To evaluate the gradient of the objective function,we derive the shape derivatives for the interfaces and apply the adjoint state method to perform the computation.The stochastic gradient descent method evaluates the gradient of the objective function only at a few samples for each iteration,which reduces the computational cost significantly.Various numerical experiments are conducted to illustrate the efficiency of the method and significant increases of the absorptance for the optimal random structures.We also examine the convergence of the stochastic gradient descent algorithm theoretically and prove that the numerical method is convergent under certain assumptions for the random interfaces.

Differentially private SGD with random features

In the realm of large-scale machine learning,it is crucial to explore methods for reducing computational complexity and memory demands while maintaining generalization performance.Additionally,since the collected data may contain some sensitive information,it is also of great significance to study privacy-preserving machine learning algorithms.This paper focuses on the performance of the differentially private stochastic gradient descent(SGD)algorithm based on random features.To begin,the algorithm maps the original data into a lowdimensional space,thereby avoiding the traditional kernel method for large-scale data storage requirement.Subsequently,the algorithm iteratively optimizes parameters using the stochastic gradient descent approach.Lastly,the output perturbation mechanism is employed to introduce random noise,ensuring algorithmic privacy.We prove that the proposed algorithm satisfies the differential privacy while achieving fast convergence rates under some mild conditions.

Fine-Tuning Cyber Security Defenses: Evaluating Supervised Machine Learning Classifiers for Windows Malware Detection

Malware attacks on Windows machines pose significant cybersecurity threats,necessitating effective detection and prevention mechanisms.Supervised machine learning classifiers have emerged as promising tools for malware detection.However,there remains a need for comprehensive studies that compare the performance of different classifiers specifically for Windows malware detection.Addressing this gap can provide valuable insights for enhancing cybersecurity strategies.While numerous studies have explored malware detection using machine learning techniques,there is a lack of systematic comparison of supervised classifiers for Windows malware detection.Understanding the relative effectiveness of these classifiers can inform the selection of optimal detection methods and improve overall security measures.This study aims to bridge the research gap by conducting a comparative analysis of supervised machine learning classifiers for detecting malware on Windows systems.The objectives include Investigating the performance of various classifiers,such as Gaussian Naïve Bayes,K Nearest Neighbors(KNN),Stochastic Gradient Descent Classifier(SGDC),and Decision Tree,in detecting Windows malware.Evaluating the accuracy,efficiency,and suitability of each classifier for real-world malware detection scenarios.Identifying the strengths and limitations of different classifiers to provide insights for cybersecurity practitioners and researchers.Offering recommendations for selecting the most effective classifier for Windows malware detection based on empirical evidence.The study employs a structured methodology consisting of several phases:exploratory data analysis,data preprocessing,model training,and evaluation.Exploratory data analysis involves understanding the dataset’s characteristics and identifying preprocessing requirements.Data preprocessing includes cleaning,feature encoding,dimensionality reduction,and optimization to prepare the data for training.Model training utilizes various supervised classifiers,and their performance is evaluated using metrics such as accuracy,precision,recall,and F1 score.The study’s outcomes comprise a comparative analysis of supervised machine learning classifiers for Windows malware detection.Results reveal the effectiveness and efficiency of each classifier in detecting different types of malware.Additionally,insights into their strengths and limitations provide practical guidance for enhancing cybersecurity defenses.Overall,this research contributes to advancing malware detection techniques and bolstering the security posture of Windows systems against evolving cyber threats.

FL-EASGD:Federated Learning Privacy Security Method Based on Homomorphic Encryption

Federated learning ensures data privacy and security by sharing models among multiple computing nodes instead of plaintext data.However,there is still a potential risk of privacy leakage,for example,attackers can obtain the original data through model inference attacks.Therefore,safeguarding the privacy of model parameters becomes crucial.One proposed solution involves incorporating homomorphic encryption algorithms into the federated learning process.However,the existing federated learning privacy protection scheme based on homomorphic encryption will greatly reduce the efficiency and robustness when there are performance differences between parties or abnormal nodes.To solve the above problems,this paper proposes a privacy protection scheme named Federated Learning-Elastic Averaging Stochastic Gradient Descent(FL-EASGD)based on a fully homomorphic encryption algorithm.First,this paper introduces the homomorphic encryption algorithm into the FL-EASGD scheme to preventmodel plaintext leakage and realize privacy security in the process ofmodel aggregation.Second,this paper designs a robust model aggregation algorithm by adding time variables and constraint coefficients,which ensures the accuracy of model prediction while solving performance differences such as computation speed and node anomalies such as downtime of each participant.In addition,the scheme in this paper preserves the independent exploration of the local model by the nodes of each party,making the model more applicable to the local data distribution.Finally,experimental analysis shows that when there are abnormalities in the participants,the efficiency and accuracy of the whole protocol are not significantly affected.

A Primal-Dual SGD Algorithm for Distributed Nonconvex Optimization

The distributed nonconvex optimization problem of minimizing a global cost function formed by a sum of n local cost functions by using local information exchange is considered.This problem is an important component of many machine learning techniques with data parallelism,such as deep learning and federated learning.We propose a distributed primal-dual stochastic gradient descent(SGD)algorithm,suitable for arbitrarily connected communication networks and any smooth(possibly nonconvex)cost functions.We show that the proposed algorithm achieves the linear speedup convergence rate O(1/(√nT))for general nonconvex cost functions and the linear speedup convergence rate O(1/(nT)) when the global cost function satisfies the Polyak-Lojasiewicz(P-L)condition,where T is the total number of iterations.We also show that the output of the proposed algorithm with constant parameters linearly converges to a neighborhood of a global optimum.We demonstrate through numerical experiments the efficiency of our algorithm in comparison with the baseline centralized SGD and recently proposed distributed SGD algorithms.

Two-Timescale Online Learning of Joint User Association and Resource Scheduling in Dynamic Mobile Edge Computing

For the mobile edge computing network consisting of multiple base stations and resourceconstrained user devices,network cost in terms of energy and delay will incur during task offloading from the user to the edge server.With the limitations imposed on transmission capacity,computing resource,and connection capacity,the per-slot online learning algorithm is first proposed to minimize the time-averaged network cost.In particular,by leveraging the theories of stochastic gradient descent and minimum cost maximum flow,the user association is jointly optimized with resource scheduling in each time slot.The theoretical analysis proves that the proposed approach can achieve asymptotic optimality without any prior knowledge of the network environment.Moreover,to alleviate the high network overhead incurred during user handover and task migration,a two-timescale optimization approach is proposed to avoid frequent changes in user association.With user association executed on a large timescale and the resource scheduling decided on the single time slot,the asymptotic optimality is preserved.Simulation results verify the effectiveness of the proposed online learning algorithms.

Chimp Optimization Algorithm Based Feature Selection with Machine Learning for Medical Data Classification

Datamining plays a crucial role in extractingmeaningful knowledge fromlarge-scale data repositories,such as data warehouses and databases.Association rule mining,a fundamental process in data mining,involves discovering correlations,patterns,and causal structures within datasets.In the healthcare domain,association rules offer valuable opportunities for building knowledge bases,enabling intelligent diagnoses,and extracting invaluable information rapidly.This paper presents a novel approach called the Machine Learning based Association Rule Mining and Classification for Healthcare Data Management System(MLARMC-HDMS).The MLARMC-HDMS technique integrates classification and association rule mining(ARM)processes.Initially,the chimp optimization algorithm-based feature selection(COAFS)technique is employed within MLARMC-HDMS to select relevant attributes.Inspired by the foraging behavior of chimpanzees,the COA algorithm mimics their search strategy for food.Subsequently,the classification process utilizes stochastic gradient descent with a multilayer perceptron(SGD-MLP)model,while the Apriori algorithm determines attribute relationships.We propose a COA-based feature selection approach for medical data classification using machine learning techniques.This approach involves selecting pertinent features from medical datasets through COA and training machine learning models using the reduced feature set.We evaluate the performance of our approach on various medical datasets employing diverse machine learning classifiers.Experimental results demonstrate that our proposed approach surpasses alternative feature selection methods,achieving higher accuracy and precision rates in medical data classification tasks.The study showcases the effectiveness and efficiency of the COA-based feature selection approach in identifying relevant features,thereby enhancing the diagnosis and treatment of various diseases.To provide further validation,we conduct detailed experiments on a benchmark medical dataset,revealing the superiority of the MLARMCHDMS model over other methods,with a maximum accuracy of 99.75%.Therefore,this research contributes to the advancement of feature selection techniques in medical data classification and highlights the potential for improving healthcare outcomes through accurate and efficient data analysis.The presented MLARMC-HDMS framework and COA-based feature selection approach offer valuable insights for researchers and practitioners working in the field of healthcare data mining and machine learning.

Routing with Cooperative Nodes Using Improved Learning Approaches

In IoT,routing among the cooperative nodes plays an incredible role in fulfilling the network requirements and enhancing system performance.The eva-luation of optimal routing and related routing parameters over the deployed net-work environment is challenging.This research concentrates on modelling a memory-based routing model with Stacked Long Short Term Memory(s-LSTM)and Bi-directional Long Short Term Memory(b-LSTM).It is used to hold the routing information and random routing to attain superior performance.The pro-posed model is trained based on the searching and detection mechanisms to com-pute the packet delivery ratio(PDR),end-to-end(E2E)delay,throughput,etc.The anticipated s-LSTM and b-LSTM model intends to ensure Quality of Service(QoS)even in changing network topology.The performance of the proposed b-LSTM and s-LSTM is measured by comparing the significance of the model with various prevailing approaches.Sometimes,the performance is measured with Mean Absolute Error(MAE)and Root Mean Square Error(RMSE)for mea-suring the error rate of the model.The prediction of error rate is made with Learn-ing-based Stochastic Gradient Descent(L-SGD).This gradual gradient descent intends to predict the maximal or minimal error through successive iterations.The simulation is performed in a MATLAB 2020a environment,and the model performance is evaluated with diverse approaches.The anticipated model intends to give superior performance in contrast to prevailing approaches.

Effective and Efficient Video Compression by the Deep Learning Techniques

Deep learning has reached many successes in Video Processing.Video has become a growing important part of our daily digital interactions.The advancement of better resolution content and the large volume offers serious challenges to the goal of receiving,distributing,compressing and revealing highquality video content.In this paper we propose a novel Effective and Efficient video compression by the Deep Learning framework based on the flask,which creatively combines the Deep Learning Techniques on Convolutional Neural Networks(CNN)and Generative Adversarial Networks(GAN).The video compression method involves the layers are divided into different groups for data processing,using CNN to remove the duplicate frames,repeating the single image instead of the duplicate images by recognizing and detecting minute changes using GAN and recorded with Long Short-Term Memory(LSTM).Instead of the complete image,the small changes generated using GAN are substituted,which helps with frame-level compression.Pixel wise comparison is performed using K-nearest Neighbours(KNN)over the frame,clustered with K-means and Singular Value Decomposition(SVD)is applied for every frame in the video for all three colour channels[Red,Green,Blue]to decrease the dimension of the utility matrix[R,G,B]by extracting its latent factors.Video frames are packed with parameters with the aid of a codec and converted to video format and the results are compared with the original video.Repeated experiments on several videos with different sizes,duration,Frames per second(FPS),and quality results demonstrated a significant resampling rate.On normal,the outcome delivered had around a 10%deviation in quality and over half in size when contrasted,and the original video.

An anti-main-lobe jamming algorithm for airborne early warning radar based on APC-SVRGD joint optimization

Main lobe jamming seriously affects the detection performance of airborne early warning radar.The joint processing of polarization-space has become an effective way to suppress the main lobe jamming.To avoid the main beam distortion and wave crest migration caused by the main lobe jamming in adaptive beamforming,a joint optimization algorithm based on adaptive polarization canceller(APC)and stochastic variance reduction gradient descent(SVRGD)is proposed.First,the polarization plane array structure and receiving signal model based on primary and auxiliary array cancellation are established,and an APC iterative algorithm model is constructed to calculate the optimal weight vector of the auxiliary channel.Second,based on the stochastic gradient descent principle,the variance reduction method is introduced to modify the gradient through internal and external iteration to reduce the variance of the stochastic gradient estimation,the airspace optimal weight vector is calculated and the equivalent weight vector is introduced to measure the beamforming effect.Third,by setting up a planar polarization array simulation scene,the performance of the algorithm against the interference of the main lobe and the side lobe is analyzed,and the effectiveness of the algorithm is verified under the condition of short snapshot number and certain signal to interference plus noise ratio.

Numerical and experimental study on coherent beam combining of fibre amplifiers using simulated annealing algorithm

We present the numerical and experimental study on the coherent beam combining of fibre amplifiers by means of simulated annealing （SA） algorithm. The feasibility is validated by the Monte Carlo simulation of correcting static phase distortion using SA algorithm. The performance of SA algorithm under time-varying phase noise is numerically studied by dynamic simulation. It is revealed that the influence of phase noise on the performance of SA algorithm gets stronger with an increase in amplitude or frequency of phase noise; and the laser array that contains more lasers will be more affected from phase noise. The performance of SA algorithm for coherent beam combining is also compared with a widely used stochastic optimization algorithm, i.e., the stochastic parallel gradient descent （SPGD） algorithm. In a proof-of-concept experiment we demonstrate the coherent beam combining of two 1083~nm fibre amplifiers with a total output power of 12~W and 93% combining efficiency. The contrast of the far-field coherently combined beam profiles is calculated to be as high as 95%.

Coherent beam combining of hybrid phase control in master oscillator-power amplifier configuration

A novel scalable architecture for coherent beam combining with hybrid phase control involving passive phasing and active phasing in master oscillator-power amplifier configuration is presented. Wide-linewidth mutually injected passive phasing fibre laser arrays serve as master oscillators for the power amplifiers, and the active phasing using stochastic parallel gradient descent algorithm is induced. Wide-linewidth seed laser can suppress the stimulated Brillouin scattering effectively and improve the output power of the fibre laser amplifier, while hybrid phase control provides a robust way for in-phase mode coherent beam combining simultaneously. Experiment is performed by active phasing fibre laser amplifiers with passive phasing fibre ring laser array seed lasers. Power encircled in the main-lobe increases1.57 times and long-exposure fringe contrast is obtained to be 78% when the system evolves from passive phasing to hybrid phasing.

Adam revisited:a weighted past gradients perspective

Adaptive learning rate methods have been successfully applied in many fields,especially in training deep neural networks.Recent results have shown that adaptive methods with exponential increasing weights on squared past gradients(i.e.,ADAM,RMSPROP)may fail to converge to the optimal solution.Though many algorithms,such as AMSGRAD and ADAMNC,have been proposed to fix the non-convergence issues,achieving a data-dependent regret bound similar to or better than ADAGRAD is still a challenge to these methods.In this paper,we propose a novel adaptive method weighted adaptive algorithm(WADA)to tackle the non-convergence issues.Unlike AMSGRAD and ADAMNC,we consider using a milder growing weighting strategy on squared past gradient,in which weights grow linearly.Based on this idea,we propose weighted adaptive gradient method framework(WAGMF)and implement WADA algorithm on this framework.Moreover,we prove that WADA can achieve a weighted data-dependent regret bound,which could be better than the original regret bound of ADAGRAD when the gradients decrease rapidly.This bound may partially explain the good performance of ADAM in practice.Finally,extensive experiments demonstrate the effectiveness of WADA and its variants in comparison with several variants of ADAM on training convex problems and deep neural networks.

Research on three-step accelerated gradient algorithm in deep learning

Gradient descent(GD)algorithm is the widely used optimisation method in training machine learning and deep learning models.In this paper,based on GD,Polyak’s momentum(PM),and Nesterov accelerated gradient(NAG),we give the convergence of the algorithms from an ini-tial value to the optimal value of an objective function in simple quadratic form.Based on the convergence property of the quadratic function,two sister sequences of NAG’s iteration and par-allel tangent methods in neural networks,the three-step accelerated gradient(TAG)algorithm is proposed,which has three sequences other than two sister sequences.To illustrate the perfor-mance of this algorithm,we compare the proposed algorithm with the three other algorithms in quadratic function,high-dimensional quadratic functions,and nonquadratic function.Then we consider to combine the TAG algorithm to the backpropagation algorithm and the stochastic gradient descent algorithm in deep learning.For conveniently facilitate the proposed algorithms,we rewite the R package‘neuralnet’and extend it to‘supneuralnet’.All kinds of deep learning algorithms in this paper are included in‘supneuralnet’package.Finally,we show our algorithms are superior to other algorithms in four case studies.

A Sample-Wise Data Driven Control Solver for the Stochastic Optimal Control Problem with Unknown Model Parameters

In this work,an efficient sample-wise data driven control solver will be developed to solve the stochastic optimal control problem with unknown model parameters.A direct filter method will be applied as an online parameter estimation method that dynamically estimates the target model parameters upon receiving the data,and a sample-wise optimal control solver will be provided to efficiently search for the optimal control.Then,an effective overarching algorithm will be introduced to combine the parameter estimator and the optimal control solver.Numerical experiments will be carried out to demonstrate the effectiveness and the efficiency of the sample-wise data driven control method.

Neural-network-based Power System State Estimation with Extended Observability

This paper proposes a neural-network-based state estimation(NNSE)method that aims to achieve higher time efficiency,improved robustness against noise,and extended observability when compared with the conventional weighted least squares(WLS)state estimation method.NNSE consists of two parts,the linear state estimation neural network(LSE-net)and the unobservable state estimation neural network(USE-net).The LSE-net functions as an adaptive approximator of linear state estimation(LSE)equations to estimate the nominally observable states.The inputs of LSE-net are the vectors of synchrophasors while the outputs are the estimated states.The USE-net operates as the complementary estimator on the nominally unobservable states.The inputs are the estimated observable states from LSE-net while the outputs are the estimation of nominally unobservable states.USE-net is trained off-line to approximate the veiled relationship between observable states and unobservable states.Two test cases are conducted to validate the performance of the proposed approach.The first case,which is based on the IEEE 118-bus system,shows the comprehensive performance of convergence,accuracy,and robustness of the proposed approach.The second case study adopts real-world synchrophasor measurements,and is based on the Jiangsu power grid,which is one of the largest provincial power systems in China.