摘要
为了解决预训练语言模型(PLMs)由于参数量过大而无法在边缘设备上运行和训练的问题,引入轻量化的迁移模块,并将迁移模块和大型预训练语言模型分离部署,实现高效的云边协同迁移学习框架.利用所提框架,可以在仅微调少量参数的条件下将大型预训练语言模型的表征迁移到下游任务,还可以进行跨领域的云边协同推理.多个领域的下游任务可以协同共享同一个预训练语言模型,能有效节省计算开销.任务可以高效地分离部署在不同的设备上,实现多个任务的分离部署和预训练模型共享.在4项公开自然语言任务数据集上进行实验验证,结果表明,该框架的性能表现能达到完全微调BERT方法的95%以上.
A lightweight transfer module was introduced to re solve the problem that current pre-trained language models(PLMs)cannot be operated and trained on edge devices due to the excessive number of parameters.The deployment of the transfer module was separated from the large PLM,and an efficient cloud-side collaborative transfer learning framework was implemented,which could transfer PLM to downstream tasks with only a small number of parameters fine-tuned.Cross-domain cloud-side collaborative deployment was also supported.Downstream tasks in multiple domain can collaboratively share the same PLM,which effectively saves computing overhead.Tasks can be efficiently separated and deployed on different devices to realize the separate deployment of multiple tasks and the sharing of PLM.Experiments on four public natural language processing task datasets were conducted,and the results showed that the performance of this framework was over 95%of that of fully fine-tuned BERT methods.
作者
赵蕴龙
赵敏喆
朱文强
查星宇
ZHAO Yunlong;ZHAO Minzhe;ZHU Wenqiang;CHA Xingyu(School of Computer Science and Technology,Nanjing University of Aeronautics and Astronautics,Nanjing 211106,China)
出处
《浙江大学学报(工学版)》
EI
CAS
CSCD
北大核心
2024年第12期2531-2539,共9页
Journal of Zhejiang University:Engineering Science
基金
国家重点研发计划资助项目(2022ZD0115403)
国家自然科学基金资助项目(62072236)。
关键词
自然语言处理
迁移学习
云边协同
计算效率
模型部署
natural language processing
transfer learning
cloud-edge collaboration
computation efficiency
model deployment