基于多级联深度学习处理器的机器人手术器械检测和姿态估计算法研究

Research on robot-based surgical instrument detection and pose estimation algorithm with multi-cascade deep learning processor

目的:为完成器械护士机器人手术器械识别及传递任务,提出一种基于多级联深度学习处理器的手术器械检测和姿态估计算法。方法:提出的多级联深度学习处理器CYSP算法级联了添加坐标注意力模块的YOLOX(YOLOX with coordinate attention block,CA-YOLOX)、分割一切模型(segment anything model,SAM)、主成分分析(principal component analysis,PCA)等功能模组。首先,应用CA-YOLOX进行器械种类识别,完成x、y坐标粗定位;其次,利用SAM分割器明确手术器械在RGB图像中的位置,引入深度信息和相机内部参数获得手术器械点云;最后,对手术器械点云使用PCA算法获得其质心、主方向和法线方向,凭此求解目标坐标系(手术器械质心坐标系)与机械臂基坐标系间的旋转平移(rotation and translation,RT)矩阵,并将该矩阵转换为四元数传递给机械臂控制单元,使机械臂可以到达相应位置拾取器械,完成器械传递任务。在自建的手术器械图像数据集上完成迁移训练并评估提出算法的效果,在七自由度机械臂上进行器械传递实验并评估该算法的成功率。结果:多级联深度学习处理器CYSP算法在手术器械数据集上的识别准确率为98.52%,器械传递实验的成功率为94%,识别平均用时为0.28 s。结论:多级联深度学习处理器CYSP算法具有较好的可靠性和实用性,能有效辅助器械护士机器人完成手术器械识别及传递任务。

Objective To propose a multi-cascade deep learning processor-based surgical instrument detection and pose estimation algorithm to facilitate the robotic scurb nurse to recognize and delivery surgical instruments.Methods The proposed multi-cascade deep leaning processor-based CYSP algorithm was hibernated with several functional modules such as YOLOX with coordinate attention block(CA-YOLOX),segment anything model(SAM)and principal component analysis(PCA).Firstly,CA-YOLOX was applied to identifying the types of the surgical instruments and completing the coarse positioning of x and y coordinates;secondly,the SAM segmenter was used to clarify the positions of the instruments in the RGB image,and the depth information and internal parameters of the camera were introduced to obtain the point cloud of the surgical instruments;finally,the center of mass,principal direction and normal direction of the surgical instrument point cloud were determined through the PCA algorithm,with which the rotation and translation(RT)matrix between the target coordinate system(surgical instrument center of mass coordinate system)and the base coordinate system of the robotic arm was solved,and the matrix was converted into a quaternion and then transmitted to the robotic arm control unit so as to drive the robotic arm to arrive at the corresponding position and pick up the instrument to complete the instrument delivery task.Migration training was accomplished on a self-constructed surgical instrument image dataset and the effectiveness of the proposed algorithm was evaluated,and instrument delivery experiments were performed on a seven-degree-of-freedom robotic arm and the success rate of the algorithm was assessed.Results The multi-cascade deep leaning processor-based CYSP algorithm had a recognition accuracy of 98.52% on the surgical instrument dataset,a success rate of 94% for the instrument delivery experiment and average time for recognition of 0.28 s.Conclusion The multi-cascade deep leaning processor-based CYSP algorithm with high reliability and practicability behaves well in facilitating the robotic scurb nurse to recognize and deliver surgical instruments.