摘要
由于视觉单模态转译技能方法容易出现错误,针对该问题,设计一种结合视觉和文本的多模态视频转译技能方法。该方法包含两个模块,第一个模块通过使用I3D网络、Mask R-CNN模型和XGBoost分类器识别视频中的动作和物体,并组合生成视觉机器人指令。第二个模块使用BERT-GRU模型从文本语料库中进行机器指令的学习,再将模块一的机器人指令输入到BERT-GRU模型上进行测试,最终生成修正过的机器人指令给机器人执行。通过在MPII Cooking 2数据集上进行实验,证明了框架性能相比于只有视频单模态情况具有较大的提升。
In order to solve the problem of visual single mode translation skills prone to recognize incorrectly,we propose a multi-modal video translation skill method that combines visual and textual knowledge.The method contained two modules.The first module adopted the I3D network,Mask R-CNN model and XGBoost classifier to recognize the actions and objects in the video,and combined them to generate visual robot commands.In the second module,the BERT-GRU model was used to learn robotic commands from the text corpus,and we inputted the robotic commands generated by the first module into the BERT-GRU model for testing.The corrected robot commands were generated for robot execution.Experiments conducted on the MPII Cooking 2 dataset show that the proposed method has achieved a better improvement compared with the one with single modal.
作者
黄可思
陈俊洪
林大润
王思涵
刘文印
Huang Kesi;Chen Junhong;Lin Darun;Wang Sihan;Liu Wenyin(School of Computers,Guangdong University of Technology,Guangzhou 510006,Guangdong,China)
出处
《计算机应用与软件》
北大核心
2023年第8期161-166,173,共7页
Computer Applications and Software
基金
国家自然科学基金项目(91748107,61703109)
广东省引进创新科研团队计划项目(2014ZT05G157)
广东省科技创新战略专项(pdjh2020a0173)。