With the advancement of deep learning techniques,the number of model parameters has been increasing,leading to significant memory consumption and limits in the deployment of such models in real-time applications.To re...With the advancement of deep learning techniques,the number of model parameters has been increasing,leading to significant memory consumption and limits in the deployment of such models in real-time applications.To reduce the number of model parameters and enhance the generalization capability of neural networks,we propose a method called Decoupled MetaDistil,which involves decoupled meta-distillation.This method utilizes meta-learning to guide the teacher model and dynamically adjusts the knowledge transfer strategy based on feedback from the student model,thereby improving the generalization ability.Furthermore,we introduce a decoupled loss method to explicitly transfer positive sample knowledge and explore the potential of negative samples knowledge.Extensive experiments demonstrate the effectiveness of our method.展开更多
基金supported by the Key R&D Program of Shandong Province,China(2022CXGC20106)the Pilot Project for Integrated Innovation of Science,Education,and Industry of Qilu University of Technology(Shandong Academy of Sciences)(2022JBZ01-01)+1 种基金Joint Fund of Shandong Natural Science Foundation(ZR2022LZH010)Shandong Provincial Natural Science Foundation(ZR2021LZH008).
文摘With the advancement of deep learning techniques,the number of model parameters has been increasing,leading to significant memory consumption and limits in the deployment of such models in real-time applications.To reduce the number of model parameters and enhance the generalization capability of neural networks,we propose a method called Decoupled MetaDistil,which involves decoupled meta-distillation.This method utilizes meta-learning to guide the teacher model and dynamically adjusts the knowledge transfer strategy based on feedback from the student model,thereby improving the generalization ability.Furthermore,we introduce a decoupled loss method to explicitly transfer positive sample knowledge and explore the potential of negative samples knowledge.Extensive experiments demonstrate the effectiveness of our method.
