基于特征调节器和双路径引导的RGB-D室内语义分割

RGB-D indoor semantic segmentation based on feature regulator and dual-path guidance

针对室内场景图像语义分割结果不精确、显著图粗糙的问题,提出一种基于多模态特征优化提取和双路径引导解码的网络架构(feature regulator and dual-path guidance,FG-Net)。具体来说,设计的特征调节器对每个阶段的多模态特征依次进行噪声过滤、重加权表示、差异性互补和交互融合,通过强化RGB和深度特征聚合,优化特征提取过程中的多模态特征表示。然后,在解码阶段引入特征交互融合后丰富的跨模态线索,进一步发挥多模态特征的优势。结合双路径协同引导结构,在解码阶段融合多尺度、多层次的特征信息,从而输出更细致的显著图。实验在公开数据集NYUD-v2和SUN RGB-D上进行,在主要评价指标mIoU上达到48.5%,优于其他先进算法。结果表明,该算法实现了更精细的室内场景图像语义分割,表现出了较好的泛化性和鲁棒性。

Aiming at the problems of inaccurate semantic segmentation results and rough saliency maps of indoor scene images,this paper proposed a network architecture(feature regulator and dual-path guidance,FG-Net)based on multi-modal feature optimization extraction and dual-path guided decoding.Specifically,the feature regulator sequentially performed noise filtering,re-weighted representation,differential complementation and interactive fusion on the multi-modal features at each stage,and optimized multi-modal feature representation in the feature extraction process by strengthening RGB and depth feature aggregation.Then,the dual-path guidance component introduced rich cross-modal cues after feature interactive fusion in the decoding stage to further take advantage of multi-modal features.The dual-path cooperative guidance structure outputted a more detailed saliency map by integrating multi-scale and multi-level feature information in the decoding stage.This paper conducted experiments on the public datasets NYUD-v2 and SUN RGB-D,and achieved 48.5%in the main evaluation metric mIoU,which is better than other state-of-the-art algorithms.The results show that the algorithm achieves more refined semantic segmentation of indoor scene images,and has good generalization and robustness.