基于交叉层级数据共享的多任务模型

Cross-layer data sharing based multi-task model

针对多任务学习模型中相关度低的任务之间存在的负迁移现象和信息共享困难问题,提出了一种基于交叉层级数据共享的多任务模型。该模型关注细粒度的知识共享,且能保留浅层共享专家的记忆能力和深层特定任务专家的泛化能力。首先,统一多层级共享专家,以获取复杂相关任务间的公共知识;然后,将共享信息分别迁移到不同层级的特定任务专家之中,从而在上下层之间共享部分公共知识;最后,利用基于数据样本的门控网络自主选择不同任务所需信息,从而减轻样本依赖性对模型的不利影响。相较于多门控混合专家(MMOE)模型,所提模型在UCI census-income数据集上对两个任务的F1值分别提高了7.87个百分点和1.19个百分点;且在MovieLens数据集上的回归任务的均方误差(MSE)值降低到0.0047,分类任务的AUC值提高到0.642。实验结果表明,所提出的模型适用于改善负迁移现象的影响,且能更高效地学习复杂相关任务之间的公共信息。

To address the issues of negative transfer and difficulty of information sharing between loosely correlated tasks in multi-task learning model,a cross-layer data sharing based multi-task model was proposed.The proposed model pays attention to fine-grained knowledge sharing,and is able to retain the memory ability of shallow layer shared experts and generalization ability of deep layer specific task experts.Firstly,multi-layer shared experts were unified to obtain public knowledge among complicatedly correlated tasks.Then,the shared information was transferred to specific task experts at different layers for sharing partial public knowledge between the upper and lower layers.Finally,the data sample based gated network was used to select the needed information for different tasks autonomously,thereby alleviating the harmful effects of sample dependence to the model.Compared with the Multi-gate Mixture-Of-Experts(MMOE)model,the proposed model improved the F1-score of two tasks by 7.87 percentage points and 1.19 percentage points respectively on UCI censusincome dataset.The proposed model also decreased the Mean Square Error(MSE)value of regression task to 0.0047 and increased the Area Under Curve(AUC)value of classification task to 0.642 on MovieLens dataset.Experimental results demonstrate that the proposed model is suitable to improve the influence of negative transfer and can learn public information among complicated related tasks more efficiently.