On the direct searches for non-smooth stochastic optimization problems

Many difficult engineering problems cannot be solved by the conventional optimization techniques in practice. Direct searches that need no recourse to explicit derivatives are revived and become popular since the new century. In order to get a deep insight into this field, some notes on the direct searches for non-smooth optimization problems are made. The global convergence vs. local convergence and their influences on expected solutions for simulation-based stochastic optimization are pointed out. The sufficient and simple decrease criteria for step acceptance are analyzed, and why simple decrease is enough for globalization in direct searches is identified. The reason to introduce the positive spanning set and its usage in direct searches is explained. Other topics such as the generalization of direct searches to bound, linear and non-linear constraints are also briefly discussed.

Non-Convex Optimization of Resource Allocation in Fog Computing Using Successive Approximation

Fog computing can deliver low delay and advanced IT services to end users with substantially reduced energy consumption.Nevertheless,with soaring demands for resource service and the limited capability of fog nodes,how to allocate and manage fog computing resources properly and stably has become the bottleneck.Therefore,the paper investigates the utility optimization-based resource allocation problem between fog nodes and end users in fog computing.The authors first introduce four types of utility functions due to the diverse tasks executed by end users and build the resource allocation model aiming at utility maximization.Then,for only the elastic tasks,the convex optimization method is applied to obtain the optimal results;for the elastic and inelastic tasks,with the assistance of Jensen’s inequality,the primal non-convex model is approximated to a sequence of equivalent convex optimization problems using successive approximation method.Moreover,a two-layer algorithm is proposed that globally converges to an optimal solution of the original problem.Finally,numerical simulation results demonstrate its superior performance and effectiveness.Comparing with other works,the authors emphasize the analysis for non-convex optimization problems and the diversity of tasks in fog computing resource allocation.

Almost Sure Convergence of Proximal Stochastic Accelerated Gradient Methods

Proximal gradient descent and its accelerated version are resultful methods for solving the sum of smooth and non-smooth problems. When the smooth function can be represented as a sum of multiple functions, the stochastic proximal gradient method performs well. However, research on its accelerated version remains unclear. This paper proposes a proximal stochastic accelerated gradient (PSAG) method to address problems involving a combination of smooth and non-smooth components, where the smooth part corresponds to the average of multiple block sums. Simultaneously, most of convergence analyses hold in expectation. To this end, under some mind conditions, we present an almost sure convergence of unbiased gradient estimation in the non-smooth setting. Moreover, we establish that the minimum of the squared gradient mapping norm arbitrarily converges to zero with probability one.

Local Discontinuous Galerkin Methods with Novel Basis for Fractional Diffusion Equations with Non-smooth Solutions

In this paper,we develop novel local discontinuous Galerkin(LDG)methods for fractional diffusion equations with non-smooth solutions.We consider such problems,for which the solutions are not smooth at boundary,and therefore the traditional LDG methods with piecewise polynomial solutions suffer accuracy degeneracy.The novel LDG methods utilize a solution information enriched basis,simulate the problem on a paired special mesh,and achieve optimal order of accuracy.We analyze the L2 stability and optimal error estimate in L2-norm.Finally,numerical examples are presented for validating the theoretical conclusions.

IMPLICIT DETERMINANT METHOD FOR SOLVING AN HERMITIAN EIGENVALUE OPTIMIZATION PROBLEM

Implicit determinant method is an effective method for some linear eigenvalue optimization problems since it solves linear systems of equations rather than eigenpairs.In this paper,we generalize the implicit determinant method to solve an Hermitian eigenvalue optimization problem for smooth case and non-smooth case.We prove that the implicit determinant method converges locally and quadratically.Numerical experiments confirm our theoretical results and illustrate the efficiency of implicit determinant method.

A Non-Convex Optimization Approach to Dynamic Coverage Problem of Multi-agent Systems in an Environment with Obstacles

In this paper,a cooperative region reconnaissance problem is investigated where a group of agents are required to fly across and detect events occur in an environment with static obstacles until an effective coverage is achieved.First,the region reconnaissance is formulated as a non-convex optimization problem.A coverage performance index with additional collision and obstacle avoidance constraints is given.Since the optimization index is an implicit function of state variables and cannot be used to compute gradients on state variables directly,an approximate optimization index is selected.Then,a non-convex optimization-based coverage algorithm is proposed to find the optimal reconnaissance location for each agent and guarantee no collisions trajectories among agents and obstacles.Finally,simulation experiments are performed to verify the effectiveness of the proposed approach.

A Positioning Method and Realization on Single Satellites in Different Orbits Using TDOA

The main geolocation technology currently used in COSPAS-SARSAT system is TDOA/FDOA or three-star TDOA,the principle is to determine the location of the signal source by using the difference in arrival time and frequency of the wireless signal between different receivers.Therefore,ground monitoring stations need to be equipped with more than two antenna receiving stations,and multiple satellites should be able to simultaneously relay the distress signal from the target source in order to achieve the geolocation function.However,when the ground receiving system has only one antenna receiving station,or the target source is in a heavily obscured environment,the ground side is unable to receive the forwarded signals from multiple satellites at the same time,which will make it impossible to locate.To address these problems,in this paper,a time-sharing single satellite geolocations method based on different orbits is proposed for the first time.This method uses one or several low-earth orbit satellites(LEO)and mediumearth orbit satellites(MEO)in the visible area,and the receiving station only needs one pair of receiving antennas to complete the positioning.It can effectively compensate for the shortcomings of the traditional TDOA using the same moment and have better positioning accuracy compared with the single satellite in the same orbit.Due to the limited experimental conditions,this paper tests the navigation satellite using different orbit time-sharing single satellite geolocations,and proves that the positioning method has high positioning accuracy and has certain promotion and application value.

Optimization for Deep Learning:An Overview

Optimization is a critical component in deep learning.We think optimization for neural networks is an interesting topic for theoretical research due to various reasons.First,its tractability despite non-convexity is an intriguing question and may greatly expand our understanding of tractable problems.Second,classical optimization theory is far from enough to explain many phenomena.Therefore,we would like to understand the challenges and opportunities from a theoretical perspective and review the existing research in this field.First,we discuss the issue of gradient explosion/vanishing and the more general issue of undesirable spectrum and then discuss practical solutions including careful initialization,normalization methods and skip connections.Second,we review generic optimization methods used in training neural networks,such as stochastic gradient descent and adaptive gradient methods,and existing theoretical results.Third,we review existing research on the global issues of neural network training,including results on global landscape,mode connectivity,lottery ticket hypothesis and neural tangent kernel.

Decentralized Control for Residential Energy Management of a Smart Users' Microgrid with Renewable Energy Exchange

This paper presents a decentralized control strategy for the scheduling of electrical energy activities of a microgrid composed of smart homes connected to a distributor and exchanging renewable energy produced by individually owned distributed energy resources. The scheduling problem is stated and solved with the aim of reducing the overall energy supply from the grid, by allowing users to exchange the surplus renewable energy and by optimally planning users' controllable loads. We assume that each smart home can both buy/sell energy from/to the grid taking into account time-varying non-linear pricing signals. Simultaneously, smart homes cooperate and may buy/sell locally harvested renewable energy from/to other smart homes. The resulting optimization problem is formulated as a non-convex non-linear programming problem with a coupling of decision variables in the constraints. The proposed solution is based on a novel heuristic iterative decentralized scheme algorithm that suitably extends the Alternating Direction Method of Multipliers to a non-convex and decentralized setting. We discuss the conditions that guarantee the convergence of the presented algorithm. Finally, the application of the proposed technique to a case study under several scenarios shows its effectiveness.

Extended finite element methods for optimal control problems governed by Poisson equation in non-convex domains

This paper analyzes two extended finite element methods(XFEMs)for linear quadratic optimal control problems governed by Poisson equation in non-convex domains.We follow the variational discretization concept to discretize the continuous problems,and apply an XFEM with a cut-off function and a classic XFEM with a fixed enrichment area to discretize the state and co-state equations.Optimal error estimates are derived for the state,co-state and control.Numerical results confirm our theoretical results.

Application of the p-Median Problem in School Allocation

This paper focus on solving the problem of optimizing students’ orientation. After four years spent in secondary school, pupils take exams and are assigned to the high school. The main difficulty of Education Department Inspection (EDI) of Dakar lies in the allocation of pupils in the suburbs. In this paper we propose an allocation model using the p-median problem. The model takes into account the distance of the standards imposed by international organizations between pupil’s home and school. The p-median problem is a location-allocation problem that takes into account the average (total) distance between demand points (pupil’s home) and facility (pupil’s school). The p-median problem is used to determine the best location to place a limited number of schools. The model has been enhanced and applied to a wide range of school location problems in suburbs. After collecting necessary numerical data to each EDI, a formulation is presented and computational results are carried out.

Blind Image Deblurring via Adaptive Optimization with Flexible Sparse St rue ture Control

Blind image deblurring is a long-standing ill-posed inverse problem which aims to recover a latent sharp image given only a blurry observation.So far,existing studies have designed many effective priors w.r.t.the latent image within the maximum a posteriori(MAP)framework in order to narrow down the solution space.These non-convex priors are always integrated into the final deblurring model,which makes the optimization challenging.However,due to unknown image distribution,complex kernel structure and non-uniform noises in real-world scenarios,it is indeed challenging to explicitly design a fixed prior for all cases.Thus we adopt the idea of adaptive optimization and propose the sparse structure control(SSC)for the latent image during the optimization process.In this paper,we only formulate the necessary optiinization constraints in a lightweight MAP model with no priors.Then we develop an inexact projected gradient scheme to incorporate flexible SSC in MAP inference.Besides Zp-norm based SSC in our previous work,we also train a group of denoising convolutional neural networks(CNNs)to learn the sparse image structure automatically from the training data under different noise levels,and we show that CNNs-based SSC can achieve similar results compared with Zp-norm but are more robust to noise.Extensive experiments demonstrate that the proposed adaptive optimization scheme with two types of SSC achieves the state-of-the-art results on both synthetic data and real-world images.

Application of the invasive weed optimization algorithm to economic dispatch problems

In this paper the invasive weed optimization algorithm has been applied to a variety of economic dispatch （ED） problems. The ED problem is concerned with minimizing the fuel cost by optimally loading the electrical generators which are committed to supply a given demand. Some involve prohibited operating zones, transmission losses and valve point loading. In general, they are non-linear non-convex optimization problems which cannot be directly solved by conventional methods. In this work the invasive weed algorithm, a meta-heuristic method inspired by the proliferation of weeds, has been applied to four numerical examples and has resulted in promising solutions compared to published results.

Application of Linearized Alternating Direction Multiplier Method in Dictionary Learning

The Alternating Direction Multiplier Method (ADMM) is widely used in various fields, and different variables are customized in the literature for different application scenarios [1] [2] [3] [4]. Among them, the linearized alternating direction multiplier method (LADMM) has received extensive attention because of its effectiveness and ease of implementation. This paper mainly discusses the application of ADMM in dictionary learning (non-convex problem). Many numerical experiments show that to achieve higher convergence accuracy, the convergence speed of ADMM is slower, especially near the optimal solution. Therefore, we introduce the linearized alternating direction multiplier method (LADMM) to accelerate the convergence speed of ADMM. Specifically, the problem is solved by linearizing the quadratic term of the subproblem, and the convergence of the algorithm is proved. Finally, there is a brief summary of the full text.

A New Model for Transmission Network Expansion and Reactive Power Planning in a Deregulated Environment

Transmission network expansion planning (TNEP) is a challenging issue especially in new restructured electricity mar-kets environment. TNEP can be incorporated with reactive power planning in which the operating conditions will be satisfied. In this paper a combinatorial mathematical model has been presented to solve transmission expansion and reactive power planning problem (TEPRPP) simultaneously. The proposed model is a non-convex problem having a mixed integer nonlinear nature where the number of candidate solutions to be evaluated increases exponentially according to the system size. The objective function of TEPRPP comprises the new circuits’ investment and production costs as well as load curtailment penalty payments. A real genetic algorithm (RGA) aimed to obtaining a significant quality solution to handle such a complicated problem has been employed. An interior point method (IPM) is applied to solve the proposed concurrent optimization problem in the solution steps of TEPRPP model. This paper proposes a new methodology for the best location as well as the capacity of VAr sources;it is tested on two well-known systems;the Garver and IEEE 24-bus systems. The obtained results show the capability and the viability of the proposed TEPRPP model incorporating operating conditions.

Optimal Combined Heat and Power Economic Dispatch Using Stochastic Fractal Search Algorithm

Combined heat and power(CHP)generation is a valuable scheme for concurrent generation of electrical and thermal energies.The interdependency of power and heat productions in CHP units introduces complications and non-convexities in their modeling and optimization.This paper uses the stochastic fractal search(SFS)optimization technique to treat the highly non-linear CHP economic dispatch(CHPED)problem,where the objective is to minimize the total operation cost of both power and heat from generation units while fulfilling several operation interdependent limits and constraints.The CHPED problem has bounded feasible operation regions and many local minima.The SFS,which is a recent metaheuristic global optimization solver,outranks many current reputable solvers.Handling constraints of the CHPED is achieved by employing external penalty parameters,which penalize infeasible solution during the iterative process.To confirm the strength of this algorithm,it has been tested on two different test systems that are regularly used.The obtained outcomes are compared with former outcomes achieved by many different methods reported in literature of CHPED.The results of this work affirm that the SFS algorithm can achieve improved near-global solution and compare favorably with other commonly used global optimization techniques in terms of the quality of solution,handling of constraints and computation time.

An Efficient Smoothing and Thresholding Image Segmentation Framework with Weighted Anisotropic-lsotropicTotalVariation

In this paper,we design an efficient,multi-stage image segmentation framework that incorporates a weighted difference of anisotropic and isotropic total variation(AITV).The segmentation framework generally consists of two stages:smoothing and thresholding,thus referred to as smoothing-and-thresholding(SaT).In the first stage,a smoothed image is obtained by an AITV-regularized Mumford-Shah(MS)model,which can be solved efficiently by the alternating direction method of multipliers(ADMMs)with a closed-form solution of a proximal operator of the l_(1)-αl_(2) regularizer.The convergence of the ADMM algorithm is analyzed.In the second stage,we threshold the smoothed image by K-means clustering to obtain the final segmentation result.Numerical experiments demonstrate that the proposed segmentation framework is versatile for both grayscale and color images,effcient in producing high-quality segmentation results within a few seconds,and robust to input images that are corrupted with noise,blur,or both.We compare the AITV method with its original convex TV and nonconvex TVP(O<p<1)counterparts,showcasing the qualitative and quantitative advantages of our proposed method.

Optimization for UAV-assisted simultaneous transmission and reception communications in the existence of malicious jammers

In this paper,we study an unmanned aerial vehicle(UAV)-assisted communication system,where the UAV is dispatched to implement simultaneous transmission and reception(STR)in the existence of multiple malicious jammers.Two schemes are investigated,namely frequency band-division-duplex(FDD)and time-fraction(TF).Based on the FDD scheme,the UAV can transmit information by using the portion of the bandwidth and receive information within the remaining portion of the bandwidth simultaneously.To perform the STR within the whole bandwidth,the TF-based scheme is considered by using a fraction of a time slot for the downlink,while the remaining fraction of the time slot is allocated for the uplink.We aim to maximize the worst-case throughput by optimizing the UAV three-dimensional(3D)trajectory and resource allocation for each scheme.The optimization problem is non-convex and thus computationally intractable.To handle the nonlinear problem,we use the block coordinate decomposition method to disaggregate the optimization problem into four subproblems and adopt the successive convex approximation technique to tackle non-convex problems.The simulation results demonstrate the performance of the TF-based scheme over the benchmark schemes.

Convergence analysis of an infeasible quasi- Newton bundle method for nonsmooth convex programming

By utilizing the improvement function,we change the nonsmooth convex constrained optimization into an unconstrained optimization,and construct an infeasible quasi-Newton bundle method with proximal form.It should be noted that the objective function being minimized in unconstrained optimization subproblem may vary along the iterations(it does not change if the null step is made,otherwise it is updated to a new function).It is necessary to make some adjustment in order to obtain the convergence result.We employ the main idea of infeasible bundle method of Sagastizabal and Solodov,and under the circumstances that each iteration point may be infeasible for primal problem,we prove that each cluster point of the sequence generated by the proposed algorithm is the optimal solution to the original problem.Furthermore,for BFGS quasi-Newton algorithm with strong convex objective function,we obtain the condition which guarantees the boundedness of quasi-Newton matrices and the R-linear convergence of the iteration points.