Optimization of Random Feature Method in the High-Precision Regime

Machine learning has been widely used for solving partial differential equations(PDEs)in recent years,among which the random feature method(RFM)exhibits spectral accuracy and can compete with traditional solvers in terms of both accuracy and efficiency.Potentially,the optimization problem in the RFM is more difficult to solve than those that arise in traditional methods.Unlike the broader machine-learning research,which frequently targets tasks within the low-precision regime,our study focuses on the high-precision regime crucial for solving PDEs.In this work,we study this problem from the following aspects:(i)we analyze the coeffcient matrix that arises in the RFM by studying the distribution of singular values;(ii)we investigate whether the continuous training causes the overfitting issue;(ii)we test direct and iterative methods as well as randomized methods for solving the optimization problem.Based on these results,we find that direct methods are superior to other methods if memory is not an issue,while iterative methods typically have low accuracy and can be improved by preconditioning to some extent.

Preface Special Issues in Honor of Professor Zhong-Ci Shi(Part Ⅰ)

Professor Zhong-Ci Shi, a professor of Chinese Academy of Sciences(CAS) and world-renowned computational mathematician, passed away on 13 February, 2023. Annals of Applied Mathematics will devote a few special issues to mark his long and outstanding research career in computational mathematics.

Preface Special Issues in Honor of Professor Zhong-Ci Shi(Part Ⅱ)

Professor Zhong-Ci Shi,a professor of Chinese Academy of Sciences(CAS)and worldrenowned computational mathematician,passed away on 13 February,2023.Annals of Applied Mathematics will devote a few special issues to mark his long and outstanding research career in computational mathematics.

Preface Special Issues in Honor of Professor Zhong-Ci Shi(Part III)Professor

Professor Zhong-Ci Shi,a professor of Chinese Academy of Sciences(CAS)and worldrenowned computational mathematician,passed away on 13 February,2023.Annals of Applied Mathematics will devote a few special issues to mark his long and outstanding research career in computational mathematics.

Deep Learning-Based Numerical Methods for High-Dimensional Parabolic Partial Differential Equations and Backward Stochastic Differential Equations

We study a new algorithm for solvingparabolic partial differential equations(PDEs)and backward stochastic differential equations(BSDEs)in high dimension,which is based on an analogy between the BSDE and reinforcement learning with the gradient of the solution playing the role of the policy function,and the loss function given by the error between the prescribed terminal condition and the solution of the BSDE.The policy function is then approximated by a neural network,as is done in deep reinforcement learning.Numerical results using TensorFlow illustrate the efficiency and accuracy of the studied algorithm for several 100-dimensional nonlinear PDEs from physics and finance such as the Allen–Cahn equation,the Hamilton–Jacobi–Bellman equation,and a nonlinear pricing model for financial derivatives.

The Deep Ritz Method: A Deep Learning-Based Numerical Algorithm for Solving Variational Problems

We propose a deep learning-based method,the Deep Ritz Method,for numerically solving variational problems,particularly the ones that arise from par-tial differential equations.The Deep Ritz Method is naturally nonlinear,naturally adaptive and has the potential to work in rather high dimensions.The framework is quite simple and fits well with the stochastic gradient descent method used in deep learning.We illustrate the method on several problems including some eigenvalue problems.

A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics

A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated.General initialization schemes as well as general regimes for the network width and training data size are considered.In the overparametrized regime,it is shown that gradient descent dynamics can achieve zero training loss exponentially fast regardless of the quality of the labels.In addition,it is proved that throughout the training process the functions represented by the neural network model are uniformly close to those of a kernel method.For general values of the network width and training data size,sharp estimates of the generalization error are established for target functions in the appropriate reproducing kernel Hilbert space.

Dedicated to Professor Chi-Wang Shu on the occasion of his 60th birthday A Proposal on Machine Learning via Dynamical Systems

Wediscuss the idea of using continuous dynamicalsystemstomodel generalhigh-dimensional nonlinear functions used in machine learning.We also discuss theconnection with deep learning.

Heterogeneous Multiscale Methods: A Review

This paper gives a systematic introduction to HMM,the heterogeneous multiscale methods,including the fundamental design principles behind the HMM philosophy and the main obstacles that have to be overcome when using HMM for a particular problem.This is illustrated by examples from several application areas,including complex fluids,micro-fluidics,solids,interface problems,stochastic problems,and statistically self-similar problems.Emphasis is given to the technical tools,such as the various constrained molecular dynamics,that have been developed,in order to apply HMM to these problems.Examples of mathematical results on the error analysis of HMM are presented.The review ends with a discussion on some of the problems that have to be solved in order to make HMM a more powerful tool.

Exponential convergence of the deep neural network approximation for analytic functions Dedicated to Professor Ta Tsien Li on the Occasion of His 80th Birthday

We prove that for analytic functions in low dimension, the convergence rate of the deep neural network approximation is exponential.

Machine learning from a continuous viewpoint,I

We present a continuous formulation of machine learning,as a problem in the calculus of variations and differential-integral equations,in the spirit of classical numerical analysis.We demonstrate that conventional machine learning models and algorithms,such as the random feature model,the two-layer neural network model and the residual neural network model,can all be recovered(in a scaled form)as particular discretizations of different continuous formulations.We also present examples of new models,such as the flow-based random feature model,and new algorithms,such as the smoothed particle method and spectral method,that arise naturally from this continuous formulation.We discuss how the issues of generalization error and implicit regularization can be studied under this framework.

Deep potentials for materials science

To fill the gap between accurate(and expensive)ab initio calculations and efficient atomistic simulations based on empirical interatomic potentials,a new class of descriptions of atomic interactions has emerged and been widely applied;i.e.machine learning potentials(MLPs).One recently developed type of MLP is the deep potential(DP)method.In this review,we provide an introduction to DP methods in computational materials science.The theory underlying the DP method is presented along with a step-by-step introduction to their development and use.We also review materials applications of DPs in a wide range of materials systems.The DP Library provides a platform for the development of DPs and a database of extant DPs.We discuss the accuracy and efficiency of DPs compared with ab initio methods and empirical potentials.

MOD-Net:A Machine Learning Approach via Model-Operator-Data Network for Solving PDEs

In this paper,we propose a machine learning approach via model-operatordata network(MOD-Net)for solving PDEs.A MOD-Net is driven by a model to solve PDEs based on operator representationwith regularization fromdata.For linear PDEs,we use a DNN to parameterize the Green’s function and obtain the neural operator to approximate the solution according to the Green’s method.To train the DNN,the empirical risk consists of the mean squared loss with the least square formulation or the variational formulation of the governing equation and boundary conditions.For complicated problems,the empirical risk also includes a fewlabels,which are computed on coarse grid points with cheap computation cost and significantly improves the model accuracy.Intuitively,the labeled dataset works as a regularization in addition to the model constraints.The MOD-Net solves a family of PDEs rather than a specific one and is much more efficient than original neural operator because few expensive labels are required.We numerically show MOD-Net is very efficient in solving Poisson equation and one-dimensional radiative transfer equation.For nonlinear PDEs,the nonlinear MOD-Net can be similarly used as an ansatz for solving nonlinear PDEs,exemplified by solving several nonlinear PDE problems,such as the Burgers equation.

Variational Boundary Conditions for Molecular Dynamics Simulations of Solids at Low Temperature

Boundary conditions for molecular dynamics simulation of crystalline solids are considered with the objective of eliminating the reflection of phonons.A variational formalism is presented to construct boundary conditions that minimize total phonon reflection.Local boundary conditions that involve a few neighbors of the boundary atoms and limited number of time steps are found using the variational formalism.Their effects are studied and compared with other boundary conditions such as truncated exact boundary conditions or by appending border atoms where artificial damping forces are applied.In general it is found that,with the same cost or complexity,the variational boundary conditions perform much better than the truncated exact boundary conditions or by appending border atoms with empirical damping profiles.Practical issues of implementation are discussed for real crystals.Application to brittle fracture dynamics is illustrated.

Preface

This special issue of SCIENCE CHINA Mathematics commemorates the 8th International Congress on Industrial and Applied Mathematics （ICIAM） held at China National Convention Center, Beijing, China from August I0 to 14, 2015. More than 3100 applied mathematicians from over 70 countries and regions attended the premier international congress in the field of applied and industrial mathematics. The twelve articles included in this special issue are a snapshot of research in the field of applied math- ematics, numerical analysis and scientific computing, as well as applications in science and engineering. We believe these articles demonstrate the state of research and ever-increasing of the field in the understanding and solution of challenging problems arising in a wide range of science and engineering areas, as well as the need for continuing research. The first group of six articles covers mathematical theory and numerical approximations of partial differential equations：

Preface

This special issue is dedicated to Professor Shi Zhong-Ci on the occasion of his 80th birthday. Professor Shi was born on December 5, 1933, in Ningbo, Zhejiang Province. He completed the un- dergraduate study in Fudan University in 1955 and had studied computational mathematics in Steklov Institute of Mathematics in Moscow in 1956-1960. Professor Shi has made profound and significant con- tributions in many areas of scientific computing including finite element methods, domain decomposition methods, and multigrid methods. His achievements in the development of nonconforming finite element methods have been particularly recognized by the community.

Asymptotic Analysis of Quantum Dynamics in Crystals: the Bloch-Wigner Transform, Bloch Dynamics and Berry Phase

We study the semi-classical limit of the Schro¨dinger equation in a crystal in the presence of an external potential and magnetic field. We first introduce the Bloch-Wigner transform and derive the asymptotic equations governing this transform in the semi-classical setting. For the second part, we focus on the appearance of the Berry curvature terms in the asymptotic equations. These terms play a crucial role in many important physical phenomena such as the quantum Hall effect. We give a simple derivation of these terms in different settings using asymptotic analysis.

Effective Maxwell Equations from Time-dependent Density Functional Theory

The behavior of interacting electrons in a perfect crystal under macroscopic external electric and magnetic fields is studied. Effective Maxwell equations for the macroscopic electric and magnetic fields are derived starting from time-dependent density functional theory. Effective permittivity and permeability coefficients are obtained.

The Elastic Continuum Limit of the Tight Binding Model

The authors consider the simplest quantum mechanics model of solids, the tight binding model, and prove that in the continuum limit, the energy of tight binding model converges to that of the continuum elasticity model obtained using Cauchy-Born rule. The technique in this paper is based mainly on spectral perturbation theory for large matrices.

Effectiveness of Implicit Methods for Stiff Stochastic Differential Equations

In this paper we study the behavior of a family of implicit numerical methods applied to stochastic differential equations with multiple time scales.We show by a combination of analytical arguments and numerical examples that implicit methods in general fail to capture the effective dynamics at the slow time scale.This is due to the fact that such implicit methods cannot correctly capture non-Dirac invariant distributions when the time step size is much larger than the relaxation time of the system.