Physics-informed neural networks with residual/gradient-based adaptive sampling methods for solving partial differential equations with sharp solutions

We consider solving the forward and inverse partial differential equations(PDEs)which have sharp solutions with physics-informed neural networks(PINNs)in this work.In particular,to better capture the sharpness of the solution,we propose the adaptive sampling methods(ASMs)based on the residual and the gradient of the solution.We first present a residual only-based ASM denoted by ASMⅠ.In this approach,we first train the neural network using a small number of residual points and divide the computational domain into a certain number of sub-domains,then we add new residual points in the sub-domain which has the largest mean absolute value of the residual,and those points which have the largest absolute values of the residual in this sub-domain as new residual points.We further develop a second type of ASM(denoted by ASMⅡ)based on both the residual and the gradient of the solution due to the fact that only the residual may not be able to efficiently capture the sharpness of the solution.The procedure of ASMⅡis almost the same as that of ASMⅠ,and we add new residual points which have not only large residuals but also large gradients.To demonstrate the effectiveness of the present methods,we use both ASMⅠand ASMⅡto solve a number of PDEs,including the Burger equation,the compressible Euler equation,the Poisson equation over an Lshape domain as well as the high-dimensional Poisson equation.It has been shown from the numerical results that the sharp solutions can be well approximated by using either ASMⅠor ASMⅡ,and both methods deliver much more accurate solutions than the original PINNs with the same number of residual points.Moreover,the ASMⅡalgorithm has better performance in terms of accuracy,efficiency,and stability compared with the ASMⅠalgorithm.This means that the gradient of the solution improves the stability and efficiency of the adaptive sampling procedure as well as the accuracy of the solution.Furthermore,we also employ the similar adaptive sampling technique for the data points of boundary conditions(BCs)if the sharpness of the solution is near the boundary.The result of the L-shape Poisson problem indicates that the present method can significantly improve the efficiency,stability,and accuracy.

Fuzzy Difference Equations in Diagnoses of Glaucoma from Retinal Images Using Deep Learning

The intuitive fuzzy set has found important application in decision-making and machine learning.To enrich and utilize the intuitive fuzzy set,this study designed and developed a deep neural network-based glaucoma eye detection using fuzzy difference equations in the domain where the retinal images converge.Retinal image detections are categorized as normal eye recognition,suspected glaucomatous eye recognition,and glaucomatous eye recognition.Fuzzy degrees associated with weighted values are calculated to determine the level of concentration between the fuzzy partition and the retinal images.The proposed model was used to diagnose glaucoma using retinal images and involved utilizing the Convolutional Neural Network(CNN)and deep learning to identify the fuzzy weighted regularization between images.This methodology was used to clarify the input images and make them adequate for the process of glaucoma detection.The objective of this study was to propose a novel approach to the early diagnosis of glaucoma using the Fuzzy Expert System(FES)and Fuzzy differential equation(FDE).The intensities of the different regions in the images and their respective peak levels were determined.Once the peak regions were identified,the recurrence relationships among those peaks were then measured.Image partitioning was done due to varying degrees of similar and dissimilar concentrations in the image.Similar and dissimilar concentration levels and spatial frequency generated a threshold image from the combined fuzzy matrix and FDE.This distinguished between a normal and abnormal eye condition,thus detecting patients with glaucomatous eyes.

Meta-Auto-Decoder:a Meta-Learning-Based Reduced Order Model for Solving Parametric Partial Differential Equations

Many important problems in science and engineering require solving the so-called parametric partial differential equations(PDEs),i.e.,PDEs with different physical parameters,boundary conditions,shapes of computational domains,etc.Typical reduced order modeling techniques accelerate the solution of the parametric PDEs by projecting them onto a linear trial manifold constructed in the ofline stage.These methods often need a predefined mesh as well as a series of precomputed solution snapshots,and may struggle to balance between the efficiency and accuracy due to the limitation of the linear ansatz.Utilizing the nonlinear representation of neural networks(NNs),we propose the Meta-Auto-Decoder(MAD)to construct a nonlinear trial manifold,whose best possible performance is measured theoretically by the decoder width.Based on the meta-learning concept,the trial manifold can be learned in a mesh-free and unsupervised way during the pre-training stage.Fast adaptation to new(possibly heterogeneous)PDE parameters is enabled by searching on this trial manifold,and optionally fine-tuning the trial manifold at the same time.Extensive numerical experiments show that the MAD method exhibits a faster convergence speed without losing the accuracy than other deep learning-based methods.

Meshfree-based physics-informed neural networks for the unsteady Oseen equations

We propose the meshfree-based physics-informed neural networks for solving the unsteady Oseen equations.Firstly,based on the ideas of meshfree and small sample learning,we only randomly select a small number of spatiotemporal points to train the neural network instead of forming a mesh.Specifically,we optimize the neural network by minimizing the loss function to satisfy the differential operators,initial condition and boundary condition.Then,we prove the convergence of the loss function and the convergence of the neural network.In addition,the feasibility and effectiveness of the method are verified by the results of numerical experiments,and the theoretical derivation is verified by the relative error between the neural network solution and the analytical solution.

LaNets:Hybrid Lagrange Neural Networks for Solving Partial Differential Equations

We propose new hybrid Lagrange neural networks called LaNets to predict the numerical solutions of partial differential equations.That is,we embed Lagrange interpolation and small sample learning into deep neural network frameworks.Concretely,we first perform Lagrange interpolation in front of the deep feedforward neural network.The Lagrange basis function has a neat structure and a strong expression ability,which is suitable to be a preprocessing tool for pre-fitting and feature extraction.Second,we introduce small sample learning into training,which is beneficial to guide themodel to be corrected quickly.Taking advantages of the theoretical support of traditional numerical method and the efficient allocation of modern machine learning,LaNets achieve higher predictive accuracy compared to the state-of-the-artwork.The stability and accuracy of the proposed algorithmare demonstrated through a series of classical numerical examples,including one-dimensional Burgers equation,onedimensional carburizing diffusion equations,two-dimensional Helmholtz equation and two-dimensional Burgers equation.Experimental results validate the robustness,effectiveness and flexibility of the proposed algorithm.

Neural network as a function approximator and its application in solving differential equations

A neural network(NN) is a powerful tool for approximating bounded continuous functions in machine learning. The NN provides a framework for numerically solving ordinary differential equations(ODEs) and partial differential equations(PDEs)combined with the automatic differentiation(AD) technique. In this work, we explore the use of NN for the function approximation and propose a universal solver for ODEs and PDEs. The solver is tested for initial value problems and boundary value problems of ODEs, and the results exhibit high accuracy for not only the unknown functions but also their derivatives. The same strategy can be used to construct a PDE solver based on collocation points instead of a mesh, which is tested with the Burgers equation and the heat equation(i.e., the Laplace equation).

A Smoothing Neural Network Algorithm for Absolute Value Equations

In this paper, we give a smoothing neural network algorithm for absolute value equations (AVE). By using smoothing function, we reformulate the AVE as a differentiable unconstrained optimization and we establish a steep descent method to solve it. We prove the stability and the equilibrium state of the neural network to be a solution of the AVE. The numerical tests show the efficient of the proposed algorithm.

DISTRIBUTED PARAMETER NEURAL NETWORKS FOR SOLVING PARTIAL DIFFERENTIAL EQUATIONS

Novel distributed parameter neural networks are proposed for solving partial differential equations, and their dynamic performances are studied in Hilbert space. The locally connected neural networks are obtained by separating distributed parameter neural networks. Two simulations are also given. Both theoretical and computed results illustrate that the distributed parameter neural networks are effective and efficient for solving partial differential equation problems.

A Novel Method for Solving Ordinary Differential Equations with Artificial Neural Networks

This research work investigates the use of Artificial Neural Network (ANN) based on models for solving first and second order linear constant coefficient ordinary differential equations with initial conditions. In particular, we employ a feed-forward Multilayer Perceptron Neural Network (MLPNN), but bypass the standard back-propagation algorithm for updating the intrinsic weights. A trial solution of the differential equation is written as a sum of two parts. The first part satisfies the initial or boundary conditions and contains no adjustable parameters. The second part involves a feed-forward neural network to be trained to satisfy the differential equation. Numerous works have appeared in recent times regarding the solution of differential equations using ANN, however majority of these employed a single hidden layer perceptron model, incorporating a back-propagation algorithm for weight updation. For the homogeneous case, we assume a solution in exponential form and compute a polynomial approximation using statistical regression. From here we pick the unknown coefficients as the weights from input layer to hidden layer of the associated neural network trial solution. To get the weights from hidden layer to the output layer, we form algebraic equations incorporating the default sign of the differential equations. We then apply the Gaussian Radial Basis function (GRBF) approximation model to achieve our objective. The weights obtained in this manner need not be adjusted. We proceed to develop a Neural Network algorithm using MathCAD software, which enables us to slightly adjust the intrinsic biases. We compare the convergence and the accuracy of our results with analytic solutions, as well as well-known numerical methods and obtain satisfactory results for our example ODE problems.

On Study of Solutions of Kac-van Moerbeke Lattice and Self-dual Network Equations

The closed form of solutions of Kac-van Moerbeke lattice and self-dual network equations are considered by proposing transformations based on Riccati equation, using symbolic computation. In contrast to the numerical computation of travelling wave solutions for differential difference equations, our method obtains exact solutions which have physical relevance.

TCAS-PINN:Physics-informed neural networks with a novel temporal causality-based adaptive sampling method

Physics-informed neural networks(PINNs)have become an attractive machine learning framework for obtaining solutions to partial differential equations(PDEs).PINNs embed initial,boundary,and PDE constraints into the loss function.The performance of PINNs is generally affected by both training and sampling.Specifically,training methods focus on how to overcome the training difficulties caused by the special PDE residual loss of PINNs,and sampling methods are concerned with the location and distribution of the sampling points upon which evaluations of PDE residual loss are accomplished.However,a common problem among these original PINNs is that they omit special temporal information utilization during the training or sampling stages when dealing with an important PDE category,namely,time-dependent PDEs,where temporal information plays a key role in the algorithms used.There is one method,called Causal PINN,that considers temporal causality at the training level but not special temporal utilization at the sampling level.Incorporating temporal knowledge into sampling remains to be studied.To fill this gap,we propose a novel temporal causality-based adaptive sampling method that dynamically determines the sampling ratio according to both PDE residual and temporal causality.By designing a sampling ratio determined by both residual loss and temporal causality to control the number and location of sampled points in each temporal sub-domain,we provide a practical solution by incorporating temporal information into sampling.Numerical experiments of several nonlinear time-dependent PDEs,including the Cahn–Hilliard,Korteweg–de Vries,Allen–Cahn and wave equations,show that our proposed sampling method can improve the performance.We demonstrate that using such a relatively simple sampling method can improve prediction performance by up to two orders of magnitude compared with the results from other methods,especially when points are limited.

Multi-head neural networks for simulating particle breakage dynamics

The breakage of brittle particulate materials into smaller particles under compressive or impact loads can be modelled as an instantiation of the population balance integro-differential equation.In this paper,the emerging computational science paradigm of physics-informed neural networks is studied for the first time for solving both linear and nonlinear variants of the governing dynamics.Unlike conventional methods,the proposed neural network provides rapid simulations of arbitrarily high resolution in particle size,predicting values on arbitrarily fine grids without the need for model retraining.The network is assigned a simple multi-head architecture tailored to uphold monotonicity of the modelled cumulative distribution function over particle sizes.The method is theoretically analyzed and validated against analytical results before being applied to real-world data of a batch grinding mill.The agreement between laboratory data and numerical simulation encourages the use of physics-informed neural nets for optimal planning and control of industrial comminution processes.

Incorporating Lasso Regression to Physics-Informed Neural Network for Inverse PDE Problem

Partial Differential Equation(PDE)is among the most fundamental tools employed to model dynamic systems.Existing PDE modeling methods are typically derived from established knowledge and known phenomena,which are time-consuming and labor-intensive.Recently,discovering governing PDEs from collected actual data via Physics Informed Neural Networks(PINNs)provides a more efficient way to analyze fresh dynamic systems and establish PEDmodels.This study proposes Sequentially Threshold Least Squares-Lasso(STLasso),a module constructed by incorporating Lasso regression into the Sequentially Threshold Least Squares(STLS)algorithm,which can complete sparse regression of PDE coefficients with the constraints of l0 norm.It further introduces PINN-STLasso,a physics informed neural network combined with Lasso sparse regression,able to find underlying PDEs from data with reduced data requirements and better interpretability.In addition,this research conducts experiments on canonical inverse PDE problems and compares the results to several recent methods.The results demonstrated that the proposed PINN-STLasso outperforms other methods,achieving lower error rates even with less data.

Domain Decomposition of an Optimal Control Problem for Semi-Linear Elliptic Equations on Metric Graphs with Application to Gas Networks

We consider optimal control problems for the flow of gas in a pipe network. The equations of motions are taken to be represented by a semi-linear model derived from the fully nonlinear isothermal Euler gas equations. We formulate an optimal control problem on a given network and introduce a time discretization thereof. We then study the well-posedness of the corresponding time-discrete optimal control problem. In order to further reduce the complexity, we consider an instantaneous control strategy. The main part of the paper is concerned with a non-overlapping domain decomposition of the semi-linear elliptic optimal control problem on the graph into local problems on a small part of the network, ultimately on a single edge.

Using Artificial Neural-Networks in Stochastic Differential Equations Based Software Reliability Growth Modeling

Due to high cost of fixing failures, safety concerns, and legal liabilities, organizations need to produce software that is highly reliable. Software reliability growth models have been developed by software developers in tracking and measuring the growth of reliability. Most of the Software Reliability Growth Models, which have been proposed, treat the event of software fault detection in the testing and operational phase as a counting process. Moreover, if the size of software system is large, the number of software faults detected during the testing phase becomes large, and the change of the number of faults which are detected and removed through debugging activities becomes sufficiently small compared with the initial fault content at the beginning of the testing phase. Therefore in such a situation, we can model the software fault detection process as a stochastic process with a continuous state space. Recently, Artificial Neural Networks (ANN) have been applied in software reliability growth prediction. In this paper, we propose an ANN based software reliability growth model based on Ito type of stochastic differential equation. The model has been validated, evaluated and compared with other existing NHPP model by applying it on actual failure/fault removal data sets cited from real software development projects. The proposed model integrated with the concept of stochastic differential equation performs comparatively better than the existing NHPP based model.

A New Method for Solving Nonlinear Partial Differential Equations Based on Liquid Time-Constant Networks

In this paper,physics-informed liquid networks(PILNs)are proposed based on liquid time-constant networks(LTC)for solving nonlinear partial differential equations(PDEs).In this approach,the network state is controlled via ordinary differential equations(ODEs).The significant advantage is that neurons controlled by ODEs are more expressive compared to simple activation functions.In addition,the PILNs use difference schemes instead of automatic differentiation to construct the residuals of PDEs,which avoid information loss in the neighborhood of sampling points.As this method draws on both the traveling wave method and physics-informed neural networks(PINNs),it has a better physical interpretation.Finally,the KdV equation and the nonlinear Schr¨odinger equation are solved to test the generalization ability of the PILNs.To the best of the authors’knowledge,this is the first deep learning method that uses ODEs to simulate the numerical solutions of PDEs.

An artificial neural network based deep collocation method for the solution of transient linear and nonlinear partial differential equations

A combined deep machine learning(DML)and collocation based approach to solve the partial differential equations using artificial neural networks is proposed.The developed method is applied to solve problems governed by the Sine–Gordon equation(SGE),the scalar wave equation and elasto-dynamics.Two methods are studied:one is a space-time formulation and the other is a semi-discrete method based on an implicit Runge–Kutta(RK)time integration.The methodology is implemented using the Tensorflow framework and it is tested on several numerical examples.Based on the results,the relative normalized error was observed to be less than 5%in all cases.

Novel Multisoliton Solutions of Some Soliton Equations

The novel multisoliton solutions for the nonlinear lumped self-dual network equations, Toda lattice and KP equation were obtained by using the Hirota direct method.

Synchronization of networked multibody systems using fundamental equation of mechanics

From the analytical dynamics point of view, this paper develops an optimal control framework to synchronize networked multibody systems using the fundamental equation of mechanics. A novel robust control law derived from the framework is then used to achieve complete synchronization of networked identical or non-identical multibody systems formulated with Lagrangian dynamics. A distinctive feature of the developed control strategy is the introduction of network structures into the control requirement. The control law consists of two components, the first describing the architecture of the network and the second denoting an active feedback control strategy. A corresponding stability analysis is performed by the algebraic graph theory. A representative network composed of ten identical or non-identical gyroscopes is used as an illustrative example. Numerical simulation of the systems with three kinds of network structures, including global coupling, nearest-neighbour, and small-world networks, is given to demonstrate effectiveness of the proposed control methodology.

Linking Structural Equation Modeling with Bayesian Network and Its Application to Coastal Phytoplankton Dynamics in the Bohai Bay

Bayesian networks （BN） have many advantages over other methods in ecological modeling, and have become an increasingly popular modeling tool. However, BN are flawed in regard to building models based on inadequate existing knowledge. To overcome this limitation, we propose a new method that links BN with structural equation modeling （SEM）. In this method, SEM is used to improve the model structure for BN. This method was used to simulate coastal phytoplankton dynamics in the Bohai Bay. We demonstrate that this hybrid approach minimizes the need for expert elicitation, generates more reasonable structures for BN models, and increases the BN model＇s accuracy and reliability. These results suggest that the inclusion of SEM for testing and verifying the theoretical structure during the initial construction stage improves the effectiveness of BN models, especially for complex eco-environment systems. The results also demonstrate that in the Bohai Bay, while phytoplankton biomass has the greatest influence on phytoplankton dynamics, the impact of nutrients on phytoplankton dynamics is larger than the influence of the physical environment in summer. Furthermore, although the Redfield ratio indicates that phosphorus should be the primary nutrient limiting factor, our results show that silicate plays the most important role in regulating phytoplankton dynamics in the Bohai Bay.