Dear Editor,This letter is concerned with developing meta-learning models for fast,stable,and effective few-shot learning across tasks over a few training samples.Nowadays,deep and reinforcement learning(RL)is widely ...Dear Editor,This letter is concerned with developing meta-learning models for fast,stable,and effective few-shot learning across tasks over a few training samples.Nowadays,deep and reinforcement learning(RL)is widely used in autonomous intelligent systems(e.g.,target recognition[1],path planning[2],and robot control[3],[4]).展开更多
Physics-informed neural networks(PINNs)are known to suffer from optimization difficulty.In this work,we reveal the connection between the optimization difficulty of PINNs and activation functions.Specifically,we show ...Physics-informed neural networks(PINNs)are known to suffer from optimization difficulty.In this work,we reveal the connection between the optimization difficulty of PINNs and activation functions.Specifically,we show that PINNs exhibit high sensitivity to activation functions when solving PDEs with distinct properties.Existing works usually choose activation functions by inefficient trial-and-error.To avoid the inefficient manual selection and to alleviate the optimization difficulty of PINNs,we introduce adaptive activation functions to search for the optimal function when solving different problems.We compare different adaptive activation functions and discuss their limitations in the context of PINNs.Furthermore,we propose to tailor the idea of learning combinations of candidate activation functions to the PINNs optimization,which has a higher requirement for the smoothness and diversity on learned functions.This is achieved by removing activation functions which cannot provide higher-order derivatives from the candidate set and incorporating elementary functions with different properties according to our prior knowledge about the PDE at hand.We further enhance the search space with adaptive slopes.The proposed adaptive activation function can be used to solve different PDE systems in an interpretable way.Its effectiveness is demonstrated on a series of benchmarks.Code is available at https://github.com/LeapLabTHU/AdaAFforPINNs.展开更多
Deep learning based semi-supervised learning(SSL)algorithms have led to promising results in recent years.However,they tend to introduce multiple tunable hyper-parameters,making them less practical in real SSL scenari...Deep learning based semi-supervised learning(SSL)algorithms have led to promising results in recent years.However,they tend to introduce multiple tunable hyper-parameters,making them less practical in real SSL scenarios where the labeled data is scarce for extensive hyper-parameter search.In this paper,we propose a novel meta-learning based SSL algorithm(Meta-Semi)that requires tuning only one additional hyper-parameter,compared with a standard supervised deep learning algorithm,to achieve competitive performance under various conditions of SSL.We start by defining a meta optimization problem that minimizes the loss on labeled data through dynamically reweighting the loss on unlabeled samples,which are associated with soft pseudo labels during training.As the meta problem is computationally intensive to solve directly,we propose an efficient algorithm to dynamically obtain the approximate solutions.We show theoretically that Meta-Semi converges to the stationary point of the loss function on labeled data under mild conditions.Empirically,Meta-Semi outperforms state-of-the-art SSL algorithms significantly on the challenging semi-supervised CIFAR-100 and STL-10 tasks,and achieves competitive performance on CIFAR-10 and SVHN.展开更多
基金supported in part by the National Key R&D Program of China(2021YFB1714800)the National Natural Science Foundation of China(62173034,61925303,62025301,62088101)the CAAI-Huawei MindSpore Open Fund。
文摘Dear Editor,This letter is concerned with developing meta-learning models for fast,stable,and effective few-shot learning across tasks over a few training samples.Nowadays,deep and reinforcement learning(RL)is widely used in autonomous intelligent systems(e.g.,target recognition[1],path planning[2],and robot control[3],[4]).
基金the support of National Natural Science Foundation of China (Nos. 22172020 and 61974020)supported by the projects of Sci & Tech planning of Zhuhai City (No. ZH22017001200032PWC)。
基金supported in part by the National Natural Science Foundation of China under Grants 62276150the Guoqiang Institute of Tsinghua University.
文摘Physics-informed neural networks(PINNs)are known to suffer from optimization difficulty.In this work,we reveal the connection between the optimization difficulty of PINNs and activation functions.Specifically,we show that PINNs exhibit high sensitivity to activation functions when solving PDEs with distinct properties.Existing works usually choose activation functions by inefficient trial-and-error.To avoid the inefficient manual selection and to alleviate the optimization difficulty of PINNs,we introduce adaptive activation functions to search for the optimal function when solving different problems.We compare different adaptive activation functions and discuss their limitations in the context of PINNs.Furthermore,we propose to tailor the idea of learning combinations of candidate activation functions to the PINNs optimization,which has a higher requirement for the smoothness and diversity on learned functions.This is achieved by removing activation functions which cannot provide higher-order derivatives from the candidate set and incorporating elementary functions with different properties according to our prior knowledge about the PDE at hand.We further enhance the search space with adaptive slopes.The proposed adaptive activation function can be used to solve different PDE systems in an interpretable way.Its effectiveness is demonstrated on a series of benchmarks.Code is available at https://github.com/LeapLabTHU/AdaAFforPINNs.
基金supported by the National Key R&D Program of China(No.2021ZD0140407)the National Natural Science Foundation of China(No.62022048)the National Defense Basic Science and Technology Strengthening Program of China.
基金supported by the National Key R&D Program of China(No.2019YFC1408703)the National Natural Science Foundation of China(No.62022048)THU-Bosch JCML,and Beijing Academy of Artificial Intelligence.
文摘Deep learning based semi-supervised learning(SSL)algorithms have led to promising results in recent years.However,they tend to introduce multiple tunable hyper-parameters,making them less practical in real SSL scenarios where the labeled data is scarce for extensive hyper-parameter search.In this paper,we propose a novel meta-learning based SSL algorithm(Meta-Semi)that requires tuning only one additional hyper-parameter,compared with a standard supervised deep learning algorithm,to achieve competitive performance under various conditions of SSL.We start by defining a meta optimization problem that minimizes the loss on labeled data through dynamically reweighting the loss on unlabeled samples,which are associated with soft pseudo labels during training.As the meta problem is computationally intensive to solve directly,we propose an efficient algorithm to dynamically obtain the approximate solutions.We show theoretically that Meta-Semi converges to the stationary point of the loss function on labeled data under mild conditions.Empirically,Meta-Semi outperforms state-of-the-art SSL algorithms significantly on the challenging semi-supervised CIFAR-100 and STL-10 tasks,and achieves competitive performance on CIFAR-10 and SVHN.
