SGT-Net: A Transformer-Based Stratified Graph Convolutional Network for 3D Point Cloud Semantic Segmentation

In recent years,semantic segmentation on 3D point cloud data has attracted much attention.Unlike 2D images where pixels distribute regularly in the image domain,3D point clouds in non-Euclidean space are irregular and inherently sparse.Therefore,it is very difficult to extract long-range contexts and effectively aggregate local features for semantic segmentation in 3D point cloud space.Most current methods either focus on local feature aggregation or long-range context dependency,but fail to directly establish a global-local feature extractor to complete the point cloud semantic segmentation tasks.In this paper,we propose a Transformer-based stratified graph convolutional network(SGT-Net),which enlarges the effective receptive field and builds direct long-range dependency.Specifically,we first propose a novel dense-sparse sampling strategy that provides dense local vertices and sparse long-distance vertices for subsequent graph convolutional network(GCN).Secondly,we propose a multi-key self-attention mechanism based on the Transformer to further weight augmentation for crucial neighboring relationships and enlarge the effective receptive field.In addition,to further improve the efficiency of the network,we propose a similarity measurement module to determine whether the neighborhood near the center point is effective.We demonstrate the validity and superiority of our method on the S3DIS and ShapeNet datasets.Through ablation experiments and segmentation visualization,we verify that the SGT model can improve the performance of the point cloud semantic segmentation.

Hand Gesture Recognition Using Appearance Features Based on 3D Point Cloud

This paper presents a method for hand gesture recognition based on 3D point cloud. Digital image processing technology is used in this research. Based on the 3D point from depth camera, the system firstly extracts some raw data of the hand. After the data segmentation and preprocessing, three kinds of appearance features are extracted, including the number of stretched fingers, the angles between fingers and the gesture region’s area distribution feature. Based on these features, the system implements the identification of the gestures by using decision tree method. The results of experiment demonstrate that the proposed method is pretty efficient to recognize common gestures with a high accuracy.

A Random Fusion of Mix 3D and Polar Mix to Improve Semantic Segmentation Performance in 3D Lidar Point Cloud

This paper focuses on the effective utilization of data augmentation techniques for 3Dlidar point clouds to enhance the performance of neural network models.These point clouds,which represent spatial information through a collection of 3D coordinates,have found wide-ranging applications.Data augmentation has emerged as a potent solution to the challenges posed by limited labeled data and the need to enhance model generalization capabilities.Much of the existing research is devoted to crafting novel data augmentation methods specifically for 3D lidar point clouds.However,there has been a lack of focus on making the most of the numerous existing augmentation techniques.Addressing this deficiency,this research investigates the possibility of combining two fundamental data augmentation strategies.The paper introduces PolarMix andMix3D,two commonly employed augmentation techniques,and presents a new approach,named RandomFusion.Instead of using a fixed or predetermined combination of augmentation methods,RandomFusion randomly chooses one method from a pool of options for each instance or sample.This innovative data augmentation technique randomly augments each point in the point cloud with either PolarMix or Mix3D.The crux of this strategy is the random choice between PolarMix and Mix3Dfor the augmentation of each point within the point cloud data set.The results of the experiments conducted validate the efficacy of the RandomFusion strategy in enhancing the performance of neural network models for 3D lidar point cloud semantic segmentation tasks.This is achieved without compromising computational efficiency.By examining the potential of merging different augmentation techniques,the research contributes significantly to a more comprehensive understanding of how to utilize existing augmentation methods for 3D lidar point clouds.RandomFusion data augmentation technique offers a simple yet effective method to leverage the diversity of augmentation techniques and boost the robustness of models.The insights gained from this research can pave the way for future work aimed at developing more advanced and efficient data augmentation strategies for 3D lidar point cloud analysis.

A LiDAR Point Clouds Dataset of Ships in a Maritime Environment

For the first time, this article introduces a LiDAR Point Clouds Dataset of Ships composed of both collected and simulated data to address the scarcity of LiDAR data in maritime applications. The collected data are acquired using specialized maritime LiDAR sensors in both inland waterways and wide-open ocean environments. The simulated data is generated by placing a ship in the LiDAR coordinate system and scanning it with a redeveloped Blensor that emulates the operation of a LiDAR sensor equipped with various laser beams. Furthermore,we also render point clouds for foggy and rainy weather conditions. To describe a realistic shipping environment, a dynamic tail wave is modeled by iterating the wave elevation of each point in a time series. Finally, networks serving small objects are migrated to ship applications by feeding our dataset. The positive effect of simulated data is described in object detection experiments, and the negative impact of tail waves as noise is verified in single-object tracking experiments. The Dataset is available at https://github.com/zqy411470859/ship_dataset.

Human-robot shared control system based on 3D point cloud and teleoperation

Owing to the constraints of unstructured environments,it is difficult to ensure safe,accurate,and smooth completion of tasks using autonomous robots.Moreover,for small-batch and customized tasks,autonomous operation requires path planning for each task,thus reducing efficiency.We propose a human-robot shared control system based on a 3D point cloud and teleoperation for a robot to assist human operators in the performance of dangerous and cumbersome tasks.The system leverages the operator’s skills and experience to deal with emergencies and perform online error correction.In this framework,a depth camera acquires the 3D point cloud of the target object to automatically adjust the end-effector orientation.The operator controls the manipulator trajectory through a teleoperation device.The force exerted by the manipulator on the object is automatically adjusted by the robot,thus reducing the workload for the operator and improving the efficiency of task execution.In addition,hybrid force/motion control is used to decouple teleoperation from force control to ensure that force and position regulation will not interfere with each other.The proposed framework was validated using the ELITE robot to perform a force control scanning task.

Building Facade Point Clouds Segmentation Based on Optimal Dual-Scale Feature Descriptors

To address the current issues of inaccurate segmentation and the limited applicability of segmentation methods for building facades in point clouds, we propose a facade segmentation algorithm based on optimal dual-scale feature descriptors. First, we select the optimal dual-scale descriptors from a range of feature descriptors. Next, we segment the facade according to the threshold value of the chosen optimal dual-scale descriptors. Finally, we use RANSAC (Random Sample Consensus) to fit the segmented surface and optimize the fitting result. Experimental results show that, compared to commonly used facade segmentation algorithms, the proposed method yields more accurate segmentation results, providing a robust data foundation for subsequent 3D model reconstruction of buildings.

A Fast Compression Framework Based on 3D Point Cloud Data for Telepresence

In this paper,a novel compression framework based on 3D point cloud data is proposed for telepresence,which consists of two parts.One is implemented to remove the spatial redundancy,i.e.,a robust Bayesian framework is designed to track the human motion and the 3D point cloud data of the human body is acquired by using the tracking 2D box.The other part is applied to remove the temporal redundancy of the 3D point cloud data.The temporal redundancy between point clouds is removed by using the motion vector,i.e.,the most similar cluster in the previous frame is found for the cluster in the current frame by comparing the cluster feature and the cluster in the current frame is replaced by the motion vector for compressing the current frame.The hrst,the B-SHOT(binary signatures of histograms orientation)descriptor is applied to represent the point feature for matching the corresponding point between two frames.The second,the K-mean algorithm is used to generate the cluster because there are a lot of unsuccessfully matched points in the current frame.The matching operation is exploited to find the corresponding clusters between the point cloud data of two frames.Finally,the cluster information in the current frame is replaced by the motion vector for compressing the current frame and the unsuccessfully matched clusters in the curren t and the motion vectors are transmit ted into the rem ote end.In order to reduce calculation time of the B-SHOT descriptor,we introduce an octree structure into the B-SHOT descriptor.In particular,in order to improve the robustness of the matching operation,we design the cluster feature to estimate the similarity bet ween two clusters.Experimen tai results have shown the bet ter performance of the proposed method due to the lower calculation time and the higher compression ratio.The proposed met hod achieves the compression ratio of 8.42 and the delay time of 1228 ms compared with the compression ratio of 5.99 and the delay time of 2163 ms in the octree-based compression method under conditions of similar distortion rate.

Automated Rock Detection and Shape Analysis from Mars Rover Imagery and 3D Point Cloud Data

A new object-oriented method has been developed for the extraction of Mars rocks from Mars rover data. It is based on a combination of Mars rover imagery and 3D point cloud data. First, Navcam or Pancam images taken by the Mars rovers are segmented into homogeneous objects with a mean-shift algorithm. Then, the objects in the segmented images are classified into small rock candidates, rock shadows, and large objects. Rock shadows and large objects are considered as the regions within which large rocks may exist. In these regions, large rock candidates are extracted through ground-plane fitting with the 3D point cloud data. Small and large rock candidates are combined and postprocessed to obtain the final rock extraction results. The shape properties of the rocks （angularity, circularity, width, height, and width-height ratio） have been calculated for subsequent ~eological studies.

基于PointNet++的邻域特征增强点云语义分割方法

随着智能驾驶、机器人导航等以点云为基础的应用蓬勃发展,点云语义分割逐渐成为研究热点。然而,现有的点云语义分割方法存在局部特征提取不充分、特征融合不完整的缺陷。针对这些不足,提出了对应的解决方案。对于局部特征提取不充分的现象,通过嵌入邻域点的坐标、方向、距离等相关信息去关联邻域点的显式特征。对于特征融合不完整的现象,提出了一种最大池化与自注意力池化相结合的混合池化方法。网络架构基于PointNet++,并结合提出的局部特征提取和融合方法,在S3DIS数据集上的实验结果表明,与基线方法PointNet++相比,各评价指标都有不同程度的提高,证实了新方法的有效性和优越性。

基于无人机3D点云的高陡斜坡岩体结构特征

岩体结构控制着岩质斜坡的变形破坏模式,但传统的结构面信息调查方法存在危险性高、难度大的缺陷。为安全、高效地获取斜坡岩体结构信息,利用无人机摄影技术开展高陡岩质斜坡的测量,构建斜坡岩体的3D点云模型,识别结构面的产状、间距、迹长等岩体结构信息。对湖北秭归卡门子湾滑坡区调查验证,斜坡共发育6组优势结构面,平均间距为0.46~1.01 m,平均迹长为0.82~12.57 m;无人机3D点云获取的岩体结构信息精度满足要求,方法高效、可行;获得的岩体结构信息为斜坡岩体结构模型建立和稳定性评价等工作提供了可靠的数据基础。

改进的3D-BoNet算法应用于点云实例分割与三维重建

为了更好地利用点云数据重建室内三维模型,本文提出了一种基于3D-BoNet-IAM算法的室内场景三维重建方法。该方法通过改进3D-BoNet算法提高点云数据的实例分割精度。针对点云数据缺失问题,提出了基于平面基元合并优化的拟合平面方法,利用拟合得到的新平面重建建筑表面模型。在S3DIS和ScanNet V2数据集上验证3D-BoNet算法的改进效果。试验结果表明,本文提出的3D-BoNet-IAM算法比原始算法分割精度提高了3.3%;对比本文建模效果与其他建模效果发现,本文方法的建模效果更准确。本文方法能够提高室内点云数据的实例分割精度,同时得到高质量的室内三维模型。

一种多层多模态融合3D目标检测方法

在自动驾驶感知系统中视觉传感器与激光雷达是关键的信息来源,但在目前的3D目标检测任务中大部分纯点云的网络检测能力都优于图像和激光点云融合的网络,现有的研究将其原因总结为图像与雷达信息的视角错位以及异构特征难以匹配,单阶段融合算法难以充分融合二者的特征.为此,本文提出一种新的多层多模态融合的3D目标检测方法:首先,前融合阶段通过在2D检测框形成的锥视区内对点云进行局部顺序的色彩信息(Red Green Blue,RGB)涂抹编码;然后将编码后点云输入融合了自注意力机制上下文感知的通道扩充PointPillars检测网络;后融合阶段将2D候选框与3D候选框在非极大抑制之前编码为两组稀疏张量,利用相机激光雷达对象候选融合网络得出最终的3D目标检测结果.在KITTI数据集上进行的实验表明,本融合检测方法相较于纯点云网络的基线上有了显著的性能提升,平均mAP提高了6.24%.

A novel method for extracting skeleton of fruit treefrom 3D point clouds

Tree skeleton could be useful to agronomy researchers because the skeleton describes the shape and topological structure of a tree.The phenomenon of organs’mutual occlusion in fruit tree canopy is usually very serious,this should result in a large amount of data missing in directed laser scanning 3D point clouds from a fruit tree.However,traditional approaches can be ineffective and problematic in extracting the tree skeleton correctly when the tree point clouds contain occlusions and missing points.To overcome this limitation,we present a method for accurate and fast extracting the skeleton of fruit tree from laser scanner measured 3D point clouds.The proposed method selects the start point and endpoint of a branch from the point clouds by user’s manual interaction,then a backward searching is used to find a path from the 3D point cloud with a radius parameter as a restriction.The experimental results in several kinds of fruit trees demonstrate that our method can extract the skeleton of a leafy fruit tree with highly accuracy.

基于多分支特征融合的车载激光雷达3D目标检测算法

该文基于多分支特征融合的3D目标检测算法将无序的点云划分为规则的体素,利用体素特征编码模块和卷积神经网络学习体素特征,再将稀疏的3D数据压缩为稠密的二维鸟瞰图,最后通过2D骨干网络的粗糙分支和精细分支对多尺度鸟瞰图特征进行深度融合。该文实现了对多尺度特征的语义信息、纹理信息和上下文信息的聚合,得到了更加精确的原始空间位置信息、物体分类、位置回归和朝向预测,在KITTI数据集上取得优异的平均精度,并在保持一定帧率的同时具有较强的稳健性。

GridNet:efficiently learning deep hierarchical representation for 3D point cloud understanding

In this paper,we propose a novel and effective approach,namely GridNet,to hierarchically learn deep representation of 3D point clouds.It incorporates the ability of regular holistic description and fast data processing in a single framework,which is able to abstract powerful features progressively in an efficient way.Moreover,to capture more accurate internal geometry attributes,anchors are inferred within local neighborhoods,in contrast to the fixed or the sampled ones used in existing methods,and the learned features are thus more representative and discriminative to local point distribution.GridNet delivers very competitive results compared with the state of the art methods in both the object classification and segmentation tasks.

基于深度学习的自动驾驶场景3D目标检测方法

针对传统PV-RCNN在点云上采样效率低下和采样精度存在偏差等问题,提出了一种基于PV-RCNN改进的3D目标检测方法。更改关键点采样策略,使得有限的关键点可以更加地聚集在proposal区域范围内,更多的编码有效前景点特征来用于后面的proposal refinement,有效产生更具有代表性的关键点。用局部特征聚合的VectorPool聚合模块取代体素集抽象和ROI网格池化模块中的集合抽象,更高效的针对稀疏和不规则点云数据进行编码。在KITTI数据集上对算法验证,结果表明:行人鸟瞰图检测,困难级别检测精度提升较为显著,达到了10.46%,整体帧率提升为33.74%,文中的方法拥有更好的检测性能。

基于点云稀疏空间特征聚合激励的单阶段3D目标检测模型

针对目前基于点云的3D目标检测中单阶段体素法存在感受野固定、特征尺度单一,导致模型对点云特征学习不够充分、模型检测效果存在瓶颈等问题,提出了一种可端对端训练的基于体素的单阶段3D目标检测模型.首先,利用多尺度稀疏空间特征聚合模块,聚合点云在不同稀疏空间尺度上的特征,使特征充分保留点云的空间信息;然后,对特征进行分层激励,通过多尺度感受野对特征进行分层学习,强化特征的表达能力,降低噪声信息对检测结果的影响;最后,将特征输入检测头进行候选框的分类和回归.在公开的自动驾驶数据集KITTI上与主流单阶段3D目标检测模型进行了对比实验,包含对3类目标共9个的难度等级目标的检测.所提模型在其中5个等级中的平均准确率有明显提升,尤其对点云稀疏的目标,表现出较好的检测效果.实验结果表明,所提模型能够充分提取点云空间信息并有效地学习点云多尺度特征.

使用中心预测-聚类的3D箱体实例分割方法

随着深度学习技术在工业领域的大量部署,应用于运输、装卸、包装、分拣等环节的自动化系统成为仓储物流行业的研究热点。针对机器人箱体拆垛场景提出一个点云中心预测-聚类网络(center prediction-clustering network,CPCN),对箱体垛进行实例分割,并计算每个箱体的上表面中心坐标。CPCN在传统的语义-实例联合分割结构的基础上,为实例分割分支设计了中心预测模块和中心强化模块。中心预测模块帮助定位实例中心以避免中心点分割错误,中心强化模块令属于同一实例的点在特征空间中向中心聚集,二者有效增强了实例特征的辨识能力。在实例特征处理部分设计的中心-实例聚类方法直接对实例特征进行距离度量来计算实例标签,大幅减少了计算时间。在箱体数据集上进行的实验表明,与现有方法相比CPCN在实例分割任务中的平均精确率最低提高了0.7个百分点,最高提高了17.2个百分点,预测实例中心的准确率达到94.4%,中心偏移量低至13.70 mm,且推理速度快于同类型的联合分割网络,对于箱体拆垛任务更有针对性,具有良好的应用价值。

基于Point Transformer v2的点云枝叶分离方法研究

准确高效的点云枝叶分离对精确计算森林树木的垂直参数至关重要。然而,当前的研究方法计算成本高,且依赖先验知识导致泛化能力不足。针对以上问题,文章提出利用基于点特征的Transformer网络进行自动化的森林场景三维点云的枝叶分离研究。该方法使用Point Transformer v2网络,首先利用网格编码模块提取可学习的局部结构关系,保留点云的几何拓扑结构;其次使用分组注意力实现多通道联合学习,降低特征的冗余度,提高计算的效率;最后构建了基于点的Transformer网络实现高精度森林树木三维点云语义分割,降低了对于先验知识的需求。使用地基激光扫描仪获取的加拿大和芬兰7个不同树种样地的三维点云数据,进行枝叶分离实验和精度评价。实验结果表明,网络整体精度(OA)为94.42%,mIoU为78.89%,能够适应不同树种、不同点云密度的森林场景的枝叶分离。