大语言模型辅助下的增强现实装配方法

Augmented Reality Assembly Method Assisted by Large Language Models

基于增强现实的装配引导系统将数字信息叠加到物理场景中,有效指导了复杂装配作业任务。然而装配环境中人与物理世界的隔阂仍然巨大,待融合到物理世界的信息需事先准备好,并且需要人工在装配过程中来触发。研究实时且无处不在的提示,成为当前增强现实环境下的复杂装配研究热点,本文提出了一种基于大语言模型(LLMs)辅助的增强现实装配方法,其核心是将LLMs作为装配过程中的另外一个大脑,提供无处不在的装配引导和工艺信息提示支持。首先,建立了LLMs辅助的增强现实装配方法体系,分析了体系的要素及关联关系。其次,面向LLMs环境,构建了匹配的工艺信息模型。随后,给出了基于LLMs的辅助引导装配方法和流程。最后,结合某线缆装配专业知识,研发了一个专业问答系统,实现了LLMs智能辅助引导,使装配合格率提升了15%,并通过多个案例验证了该方法的有效性。

Augmented reality-based assembly guidance systems superimpose digital information onto physical scenes to effectively guide complex assembly tasks.However,the gap between humans and the physical world in the assembly environment is huge,and the information to be fused into the physical world needs to be prepared in advance and triggered manually during the assembly process.The study of real-time and ubiquitous prompts is a hot research topic for complex assembly in augmented reality environment.In this paper,we propose an augmented reality assembly method based on the assistance of large language models(LLMs),the core of which is to use LLMs as another brain in the assembly process,providing ubiquitous assembly guidance and prompts support for technique information.Firstly,a system of LLMs-assisted augmented reality assembly method is established,and the elements and interrelationships of the system are analysed.Secondly,a matching process information model is constructed for LLMs environment.Then,the assisted guidance decision-making method and process based on LLMs are given.Finally,combined with a cable assembly expertise,a professional question answering system is developed to realize the smart assisted guidance of LLMs.Results show that the assembly pass rate increased by 15%and the effectiveness of the method is verifed through several cases.