基于视频和文本的机器人技能指令生成方法研究

ROBOTIC COMMANDS GENERATION METHOD BASED ON VIDEO AND TEXT

由于视觉单模态转译技能方法容易出现错误,针对该问题,设计一种结合视觉和文本的多模态视频转译技能方法。该方法包含两个模块,第一个模块通过使用I3D网络、Mask R-CNN模型和XGBoost分类器识别视频中的动作和物体,并组合生成视觉机器人指令。第二个模块使用BERT-GRU模型从文本语料库中进行机器指令的学习,再将模块一的机器人指令输入到BERT-GRU模型上进行测试,最终生成修正过的机器人指令给机器人执行。通过在MPII Cooking 2数据集上进行实验,证明了框架性能相比于只有视频单模态情况具有较大的提升。

In order to solve the problem of visual single mode translation skills prone to recognize incorrectly,we propose a multi-modal video translation skill method that combines visual and textual knowledge.The method contained two modules.The first module adopted the I3D network,Mask R-CNN model and XGBoost classifier to recognize the actions and objects in the video,and combined them to generate visual robot commands.In the second module,the BERT-GRU model was used to learn robotic commands from the text corpus,and we inputted the robotic commands generated by the first module into the BERT-GRU model for testing.The corrected robot commands were generated for robot execution.Experiments conducted on the MPII Cooking 2 dataset show that the proposed method has achieved a better improvement compared with the one with single modal.