A Linear Frequency Principle Model to Understand the Absence of Overfitting in Neural Networks

Why heavily parameterized neural networks(NNs) do not overfit the data is an important long standing open question. We propose a phenomenological model of the NN training to explain this non-overfitting puzzle. Our linear frequency principle(LFP) model accounts for a key dynamical feature of NNs: they learn low frequencies first, irrespective of microscopic details. Theory based on our LFP model shows that low frequency dominance of target functions is the key condition for the non-overfitting of NNs and is verified by experiments. Furthermore,through an ideal two-layer NN, we unravel how detailed microscopic NN training dynamics statistically gives rise to an LFP model with quantitative prediction power.

A Correction and Comments on“Multi-Scale Deep Neural Network(MscaleDNN)for Solving Poisson-Boltzmann Equation in Complex Domains.CiCP,28(5):1970–2001,2020”

This note provides a correction of a missing weight constant in the MscaleDNN formula and some comments on the performance of the corrected algorithm.

MOD-Net:A Machine Learning Approach via Model-Operator-Data Network for Solving PDEs

In this paper,we propose a machine learning approach via model-operatordata network(MOD-Net)for solving PDEs.A MOD-Net is driven by a model to solve PDEs based on operator representationwith regularization fromdata.For linear PDEs,we use a DNN to parameterize the Green’s function and obtain the neural operator to approximate the solution according to the Green’s method.To train the DNN,the empirical risk consists of the mean squared loss with the least square formulation or the variational formulation of the governing equation and boundary conditions.For complicated problems,the empirical risk also includes a fewlabels,which are computed on coarse grid points with cheap computation cost and significantly improves the model accuracy.Intuitively,the labeled dataset works as a regularization in addition to the model constraints.The MOD-Net solves a family of PDEs rather than a specific one and is much more efficient than original neural operator because few expensive labels are required.We numerically show MOD-Net is very efficient in solving Poisson equation and one-dimensional radiative transfer equation.For nonlinear PDEs,the nonlinear MOD-Net can be similarly used as an ansatz for solving nonlinear PDEs,exemplified by solving several nonlinear PDE problems,such as the Burgers equation.