Convergence of Online Gradient Method with Penalty for BP Neural Networks

Online gradient method has been widely used as a learning algorithm for training feedforward neural networks. Penalty is often introduced into the training procedure to improve the generalization performance and to decrease the magnitude of network weights. In this paper, some weight boundedness and deterministic con- vergence theorems are proved for the online gradient method with penalty for BP neural network with a hidden layer, assuming that the training samples are supplied with the network in a fixed order within each epoch. The monotonicity of the error function with penalty is also guaranteed in the training iteration. Simulation results for a 3-bits parity problem are presented to support our theoretical results.

Superconvergence of nonconforming finite element penalty scheme for Stokes problem using L^2 projection method

A modified penalty scheme is discussed for solving the Stokes problem with the Crouzeix-Raviart type nonconforming linear triangular finite element. By the L^2 projection method, the superconvergence results for the velocity and pressure are obtained with a penalty parameter larger than that of the classical penalty scheme. The numerical experiments are carried out to confirm the theoretical results.

Exactness of penalization for exact minimax penalty function method in nonconvex programming

The exact minimax penalty function method is used to solve a noncon- vex differentiable optimization problem with both inequality and equality constraints. The conditions for exactness of the penalization for the exact minimax penalty function method are established by assuming that the functions constituting the considered con- strained optimization problem are invex with respect to the same function η （with the exception of those equality constraints for which the associated Lagrange multipliers are negative these functions should be assumed to be incave with respect to η）. Thus, a threshold of the penalty parameter is given such that, for all penalty parameters exceeding this threshold, equivalence holds between the set of optimal solutions in the considered constrained optimization problem and the set of minimizer in its associated penalized problem with an exact minimax penalty function. It is shown that coercivity is not suf- ficient to prove the results.

THE CONVERGENCE OF APPROACH PENALTY FUNCTION METHOD FOR APPROXIMATE BILEVEL PROGRAMMING PROBLEM

In this paper, a new algorithm-approximate penalty function method is designed, which can be used to solve a bilevel optimization problem with linear constrained function. In this kind of bilevel optimization problem. the evaluation of the objective function is very difficult, so that only their approximate values can be obtained. This algorithm is obtained by combining penalty function method and approximation in bilevel programming. The presented algorithm is completely different from existing methods. That convergence for this algorithm is proved.

Method for electromagnetic detection satellites scheduling based on genetic algorithm with alterable penalty coefficient

The electromagnetic detection satellite （EDS） is a type of earth observation satellites （EOSs）. The Information collected by EDSs plays an important role in some fields, such as industry, science and military. The scheduling of EDSs is a complex combinatorial optimization problem. Current research mainly focuses on the scheduling of imaging satellites and SAR satellites, but little work has been done on the scheduling of EDSs for its specific characteristics. A multi-satellite scheduling model is established, in which the specific constrains of EDSs are considered, then a scheduling algorithm based on the genetic algorithm （GA） is proposed. To deal with the specific constrains of EDSs, a penalty function method is introduced. However, it is hard to determine the appropriate penalty coefficient in the penalty function. Therefore, an adaptive adjustment mechanism of the penalty coefficient is designed to solve the problem, as well as improve the scheduling results. Experimental results are used to demonstrate the correctness and practicability of the proposed scheduling algorithm.

APPLICATION OF PENALTY FUNCTION METHOD IN ISOPARANIETRIC HYBRID FINITE ELEMENT ANALYSIS

By the aid of the penalty function method, the equilibrium restriction conditions were introduced to the isoparametric hybrid finite element analysis, and the concrete application course of the penalty function method in three-dimensional isoparametdc hybrid finite element was discussed. The separated penalty parameters method and the optimal hybrid element model with penalty balance were also presented. The penalty balance method can effectively refrain the parasitical stress on the premise of no additional degrees of freedom. The numeric experiment shows that the presented element not only is effective in improving greatly the numeric calculation precision of distorted grids but also has the universality.

A Smoothing Penalty Function Method for the Constrained Optimization Problem

In this paper, an approximate smoothing approach to the non-differentiable exact penalty function is proposed for the constrained optimization problem. A simple smoothed penalty algorithm is given, and its convergence is discussed. A practical algorithm to compute approximate optimal solution is given as well as computational experiments to demonstrate its efficiency.

A Discontinuous Galerkin Method with Penalty for One-Dimensional Nonlocal Diffusion Problems

There have been many theoretical studies and numerical investigations of nonlocal diffusion(ND)problems in recent years.In this paper,we propose and analyze a new discontinuous Galerkin method for solving one-dimensional steady-state and time-dependent ND problems,based on a formulation that directly penalizes the jumps across the element interfaces in the nonlocal sense.We show that the proposed discontinuous Galerkin scheme is stable and convergent.Moreover,the local limit of such DG scheme recovers classical DG scheme for the corresponding local diff usion problem,which is a distinct feature of the new formulation and assures the asymptotic compatibility of the discretization.Numerical tests are also presented to demonstrate the eff ectiveness and the robustness of the proposed method.

Approximation-Exact Penalty Function Method for Solving a Class of Stochastic Programming

We present an approximation-exact penalty function method for solving the single stage stochastic programming problem with continuous random variable. The original problem is transformed into a determinate nonlinear programming problem with a discrete random variable sequence, which is obtained by some discrete method. We construct an exact penalty function and obtain an unconstrained optimization. It avoids the difficulty in solution by the rapid growing of the number of constraints for discrete precision. Under lenient conditions, we prove the equivalence of the minimum solution of penalty function and the solution of the determinate programming, and prove that the solution sequences of the discrete problem converge to a solution to the original problem.

Composite penalty method of a low order anisotropic nonconforming finite element for the Stokes problem

Composite penalty method of a low order anisotropic nonconforming quadrilateral finite element for the Stokes problem is presented. This method with a large penalty parameter can achieve the same accuracy as the stand method with a small penalty parameter and the convergence rate of this method is two times as that of the standard method under the condition of the same order penalty parameter. The superconvergence for velocity is established as well. The results of this paper are also valid to the most of the known nonconforming finite element methods.

A Two-Grid Technique for the Penalty Method of the Steady Navier-Stokes Equations

A two grid technique for solving the steady incompressible Navier Stokes equations in a penalty method was presented and the convergence of numerical solutions was analyzed. If a coarse size H and a fine size h satisfy H=O(h 13-s )(s=0(n=2);s=12(n=3), where n is a space dimension), this method has the same convergence accuracy as the usual finite element method. But the two grid method can save a lot of computation time for its brief calculation. Moreover, a numerical test was couducted in order to verify the correctness of above theoretical analysis.

A PENALTY FUNCTION METHOD FOR THE PRINCIPAL-AGENT PROBLEM WITH AN INFINITE NUMBER OF INCENTIVE-COMPATIBILITY CONSTRAINTS UNDER MORAL HAZARD

In this paper,we propose an iterative algorithm to find the optimal incentive mechanism for the principal-agent problem under moral hazard where the number of agent action profiles is infinite,and where there are an infinite number of results that can be observed by the principal.This principal-agent problem has an infinite number of incentive-compatibility constraints,and we transform it into an optimization problem with an infinite number of constraints called a semi-infinite programming problem.We then propose an exterior penalty function method to find the optimal solution to this semi-infinite programming and illustrate the convergence of this algorithm.By analyzing the optimal solution obtained by the proposed penalty function method,we can obtain the optimal incentive mechanism for the principal-agent problem with an infinite number of incentive-compatibility constraints under moral hazard.

Improvement of the Viscous Penalty Method for Particle-Resolved Simulations

A numerical study of the parameters controlling the viscous penalty method is investigated to better set up Particle-Resolved Direct Numerical Simulations (PR-DNS) of particulate flows. Based on this analysis, improvements of the methods are proposed in order to reach an almost second order convergence in space. The viscous penalty method is validated in Stokes regime by simulating a uniform flow past a fixed isolated cylinder. Moreover, it is also utilized in moderate Reynolds number regime for a uniform flow past a square configuration of cylinder and compared in terms of friction factor to the well-known Ergun correlation.

A GENERALIZED PENALTY METHOD FOR DIFFERENTIAL VARIATIONAL-HEMIVARIATIONAL INEQUALITIES

We consider a differential variational-hemivariational inequality with constraints,in the framework of reflexive Banach spaces.The existence of a unique mild solution of the inequality,together with its stability,was proved in[1].Here,we complete these results with existence,uniqueness and convergence results for an associated penalty-type method.To this end,we construct a sequence of perturbed differential variational-hemivariational inequalities governed by perturbed sets of constraints and penalty coefficients.We prove the unique solvability of each perturbed inequality as well as the convergence of its solution to the solution of the original inequality.Then,we consider a mathematical model which describes the equilibrium of a viscoelastic rod in unilateral contact.The weak formulation of the model is in a form of a differential variational-hemivariational inequality in which the unknowns are the displacement field and the history of the deformation.We apply our abstract penalty method in the study of this inequality and provide the corresponding mechanical interpretations.

A Space-Time Interior Penalty Discontinuous Galerkin Method for the Wave Equation

A new higher-order accurate space-time discontinuous Galerkin(DG)method using the interior penalty flux and discontinuous basis functions,both in space and in time,is pre-sented and fully analyzed for the second-order scalar wave equation.Special attention is given to the definition of the numerical fluxes since they are crucial for the stability and accuracy of the space-time DG method.The theoretical analysis shows that the DG discre-tization is stable and converges in a DG-norm on general unstructured and locally refined meshes,including local refinement in time.The space-time interior penalty DG discre-tization does not have a CFL-type restriction for stability.Optimal order of accuracy is obtained in the DG-norm if the mesh size h and the time stepΔt satisfy h≅CΔt,with C a positive constant.The optimal order of accuracy of the space-time DG discretization in the DG-norm is confirmed by calculations on several model problems.These calculations also show that for pth-order tensor product basis functions the convergence rate in the L∞and L2-norms is order p+1 for polynomial orders p=1 and p=3 and order p for polynomial order p=2.

MIXED HYBRID PENALTY FINITE ELEMENT METHOD AND ITS APPLICATION

The penalty and hybrid methods are being much used in dealing with the general incompatible element, With the penalty method convergence can always be assured, but comparatively speaking its accuracy is lower, and the condition number and sparsity are not so good. With the hybrid method, convergence can be assured only when the rank condition is satisfied. So the construction of the element is extremely limited. This paper presents the mixed hybrid penalty element method, which combines the two methods together. And it is proved theoretically that this new method is convergent, and it has the same accuracy, condition number and sparsity as the compatible element. That is to say, they are optimal to each other.Finally, a new triangle element for plate bending with nine freedom degrees is constructed with this method (three degreesof freedom are given on each corner -- one displacement and tworotations), the calculating formula of the element stiffness matrix is almost the same as that of the old triangle element for plate bending with nine degrees of freedom But it is converged to true solution with arbitrary irregrlar triangle subdivision. If the true solution u?H3 with this method the linear and quadratic rates of convergence are obtianed for three bending moments and for the displacement and two rotations respectively.

A WEIGHTED PENALTY FINITE ELEMENT METHOD FOR THE ANALYSIS OF POWER-LAW FLUID FLOW PROBLEMS

In this paper, a new finite element method for the flow analysis of the viscous incompressible power-law fluid is proposed by the use of penalty-hybrid/mixed finite element formulation and by the introduction of an alternative perturbation, which is weighted by viscosity, of the continuity equation. A numerical example is presented to exhibit the efficiency of the method.

PENALTY FUNCTION METHOD OF CONTINUUM SHAPE OPTIMIZAION

The penalty function method of continuum shape optimization and its sensitivity analysis technique are presented. A relatively simple integrated shape optimization system is developed and used to optimize the design of the inner frame shape of a three-axis test table. The result shows that the method converges well, and the system is stable and reliable.

ANALYSIS OF THREE-DIMENSIONAL UPSETTING PROCESS BY THE RIGID-PLASTIC REPRODUCING KERNEL PARTICLE METHOD

A meshless approach, called the rigid-plastic reproducing kernel particle method （RKPM）, is presented for three-dimensional （3D） bulk metal forming simulation. The approach is a combination of RKPM with the flow theory of 3D rigid-plastic mechanics. For the treatments of essential boundary conditions and incompressibility constraint, the boundary singular kernel method and the modified penalty method are utilized, respectively. The arc-tangential friction model is employed to treat the contact conditions. The compression of rectangular blocks, a typical 3D upsetting operation, is analyzed for different friction conditions and the numerical results are compared with those obtained using commercial rigid-plastic FEM （finite element method） software Deform^3D. As results show, when handling 3D plastic deformations, the proposed approach eliminates the need of expensive meshing and remeshing procedures which are unavoidable in conventional FEM and can provide results that are in good agreement with finite element predictions.

A Comparison Study of Deep Galerkin Method and Deep Ritz Method for Elliptic Problems with Different Boundary Conditions

Recent years have witnessed growing interests in solving partial differential equations by deep neural networks,especially in the high-dimensional case.Unlike classical numerical methods,such as finite difference method and finite element method,the enforcement of boundary conditions in deep neural networks is highly nontrivial.One general strategy is to use the penalty method.In the work,we conduct a comparison study for elliptic problems with four different boundary conditions,i.e.,Dirichlet,Neumann,Robin,and periodic boundary conditions,using two representative methods:deep Galerkin method and deep Ritz method.In the former,the PDE residual is minimized in the least-squares sense while the corresponding variational problem is minimized in the latter.Therefore,it is reasonably expected that deep Galerkin method works better for smooth solutions while deep Ritz method works better for low-regularity solutions.However,by a number of examples,we observe that deep Ritz method can outperform deep Galerkin method with a clear dependence of dimensionality even for smooth solutions and deep Galerkin method can also outperform deep Ritz method for low-regularity solutions.Besides,in some cases,when the boundary condition can be implemented in an exact manner,we find that such a strategy not only provides a better approximate solution but also facilitates the training process.