自然语言场景下增量知识构造与遮蔽回放策略

Incremental Knowledge Construction and Mask Replay Strategy in NLP Scenario

在增量学习中,随着增量任务的数量增多,模型在新增任务上训练后,由于数据分步偏移等一系列问题,模型对旧任务上所学到的知识发生灾难性遗忘,致使模型在旧任务上性能下降.对此,本文提出了基于知识解耦的类增量学习方法,分层次的学习不同任务共有知识与特有知识,并对这两种知识进行动态的结合,应用于下游的分类任务中.并在回放学习中运用自然语言模型的遮蔽策略,促进模型快速回忆起先前任务的知识.在自然语言处理数据集AGNews、Yelp、Amazon、DBPedia和Yahoo的类增量实验中,本文所提出的方法能有效降低模型的遗忘,提高在各个任务上的准确率等一系列指标.

In increment learning,as the number of tasks increases,the knowledge learned by the model on the old task is catastrophically forgotten after the model is trained on the new task due to a series of problems such as step-by-step data migration,resulting in the degradation of the model performance on the old task.Given this problem,a class-incremental learning method based on knowledge decoupling is proposed in this study.This method can learn the common and unique knowledge of different tasks hierarchically,combine the two kinds of knowledge dynamically,and apply them to the downstream classification tasks.Besides,the mask strategy of the natural language model is used in replay learning,which prompts the model to quickly recall the knowledge of the previous tasks.In class-incremental experiments on NLP datasets—AGNews,Yelp,Amazon,DBPedia and Yahoo,the proposed method can effectively reduce the forgetting of the model and improve the accuracy and other indicators on various tasks.