基于RGB-D信息融合和目标检测的番茄串采摘点识别定位方法

Method for recognizing and locating tomato cluster picking points based on RGB-D information fusion and target detection

采摘点的识别与定位是智能采摘的关键技术,也是实现高效、适时、无损采摘的重要保证。针对复杂背景下番茄串采摘点识别定位问题,提出基于RGB-D信息融合和目标检测的番茄串采摘点识别定位方法。通过YOLOv4目标检测算法和番茄串与对应果梗的连通关系,快速识别番茄串和可采摘果梗的感兴趣区域(Region of Interest,ROI);融合RGB-D图像中的深度信息和颜色特征识别采摘点,通过深度分割算法、形态学操作、K-means聚类算法和细化算法提取果梗图像,得到采摘点的图像坐标;匹配果梗深度图和彩色图信息,得到采摘点在相机坐标系下的精确坐标;引导机器人完成采摘任务。研究和大量现场试验结果表明,该方法可在复杂近色背景下,实现番茄串采摘点识别定位,单帧图像平均识别时间为54 ms,采摘点识别成功率为93.83%,采摘点深度误差±3 mm,满足自动采摘实时性要求。

Spatial position and coordinate points(called picking points)can widely be visualized in intelligent robots for fruit picking in mechanized modern agriculture.Recognition and location of picking points have also been the key technologies to guarantee the efficient,timely,and lossless picking during fruit harvesting.A tomato cluster can be both mature and immature tomato fruits,particularly in various shapes.Meanwhile,the color of fruit stem is similar to that of branches and leaves,while,the shape of fruit stems and petioles are similar.As such,there are large depth value errors or even a lack of depth values captured by the economical RGB-D depth camera using active stereo technology.Therefore,it is very difficult for picking robots to identify the picking points of tomato clusters in a complex planting environment.In this study,a recognition and location algorithm was proposed for the picking points of tomato clusters using RGB-D information fusion and target detection.Firstly,the Region of Interest(ROIs)of tomato clusters and stems were collected via the YOLOv4 target detection,in order to efficiently locate picking targets.Then,the ROIs of pickable stems that connected to the ripe tomato cluster were determined by screening,according to the neighbor relationship between the tomato clusters and stems.Secondly,the comprehensive segmentation was selected using RGB-D information fusion,thereby to accurately recognize the picking points of stems against the ROI color background.Specifically,the tomato clusters from the nearest row were regarded as the foreground in the RGB-D image,while the rest were assumed as the background(i.e.,noise),due mainly to only that the nearest row for picking in robots.After that,the depth information segmentation and morphological operations were combined to remove the noise in the pickable stem ROI of RGB images.Subsequently,the pickable stem edges were extracted from the stem ROI using K-means clustering,together with morphological operation and RGB color features.The center point of skeleton along the X axis was set as the picking point(x,y)in image coordinate system,especially after extracting the skeleton of stem via the thinning operation.Thirdly,the RGB image and depth map of pickable stem ROI were fused to locate the picking point.Specifically,the average depth of pickable stem was calculated using the depth information of the whole pickable stem without the noise under the mean filter.Correspondingly,an accurate depth value of picking point was obtained to compare the average with the original.Finally,the picking point was converted to the robot coordinate system from image one.Eventually,the harvesting robot implemented the picking action,according to the coordinates of picking point.A field test was also conducted to verify,where the average runtime of one image was 54 ms,while the picture resolution was 1280×720,the recognition rate of picking points was 93.83%,and the depth value error of picking point was±3 mm.Thus,the proposed algorithm can fully meet the practical requirements during field operation in harvesting robots.