深度学习实时语义分割综述

Deep learning-based real-time semantic segmentation:a survey

语义分割是计算机视觉领域的一项像素级别的感知任务,目的是为图像中的每个像素分配相应类别标签,具有广泛应用。许多语义分割网络结构复杂,计算量和参数量较大,在对高分辨率图像进行像素层次的理解时具有较大的延迟,这极大限制了其在资源受限环境下的应用,如自动驾驶、辅助医疗和移动设备等。因此,实时推理的语义分割网络得到了广泛关注。本文对深度学习中实时语义分割算法进行了全面论述和分析。1)介绍了语义分割和实时语义分割任务的基本概念、应用场景和面临问题;2)详细介绍了实时语义分割算法中常用的技术和设计,包括模型压缩技术、高效卷积神经网络(convolutional neural network,CNN)模块和高效Transformer模块;3)全面整理和归纳了现阶段的实时语义分割算法,包括单分支网络、双分支网络、多分支网络、U型网络和神经架构搜索网络5种类别的实时语义分割方法,涵盖基于CNN、基于Transformer和基于混合框架的分割网络,并分析了各类实时语义分割算法的特点和局限性;4)提供了完整的实时语义分割评价体系,包括相关数据集和评价指标、现有方法性能汇总以及领域主流方法的同设备比较,为后续研究者提供统一的比较标准;5)给出结论并分析了实时语义分割领域仍存在的挑战,对实时语义分割领域未来可能的研究方向提出了相应见解。本文提及的算法、数据集和评估指标已汇总至https://github.com/xzz777/Awesome-Real-time-Semantic-Segmentation,以便后续研究者使用。

Semantic segmentation is a fundamental task in the field of computer vision,which aims to assign a category label to each pixel in the input image.Many semantic segmentation networks have complex structures,high computational costs,and massive parameters.As a result,they introduce considerable latency when performing pixel-level scene understanding on high-resolution images.These limitations greatly restrict the applicability of these methods in resourceconstrained scenarios,such as autonomous driving,medical applications,and mobile devices.Therefore,real-time semantic segmentation methods,which produce high-precision segmentation masks with fast inference speeds,receive widespread attention.This study provides a systematic and critical review of real-time semantic segmentation algorithms based on deep learning techniques to explore the development of real-time semantic segmentation in recent years. Moreover,it covers three key aspects of real-time semantic segmentation: real-time semantic segmentation networks, mainstreamdatasets, and common evaluation indicators. In addition, this study conducts a quantitative evaluation of the realtimesemantic segmentation methods discussed and provides some insights into the future development in this field. First,semantic segmentation and real-time semantic segmentation tasks and their application scenarios and challenges are introduced.The key challenge in real-time semantic segmentation mainly lies on how to extract high-quality semantic informationwith high efficiency. Second, some preliminary knowledge for studying real-time semantic segmentation algorithms isintroduced in detail. Specifically, this study introduces four kinds of general model compression methods : network pruning,neural architecture search, knowledge distillation, and parameter quantification. It also introduces some popular efficientCNN modules in real-time semantic segmentation networks, such as MobileNet, ShuffleNet, EfficientNet, and efficientTransformer modules, such as external attention, SeaFormer, and MobileViT. Then, existing real-time semantic segmentationalgorithms are organized and summarized. In accordance with the characteristics of the overall network structure,existing works are categorized into five categories : single-branch, two-branch, multibranch, U-shaped, and neuralarchitecture search networks. Specifically, the encoder of a single-branch network is a single-branch hierarchical backbonenetwork, and its decoder is usually lightweight and does not involve complex fusion of multiscale features. The two-branchnetwork adopts a two-branch encoder structure, using one branch to capture spatial detail information and the other branchto model semantic context information. Multibranch networks are characterized by a multibranch structure in the encoderpart of the network or a network with multiresolution inputs, where the input of each resolution passes through a differentsubnetwork. The U-shaped network has a contracting encoder and an expansive decoder, which are roughly symmetrical tothe encoder. Most works of these aforementioned four categories are manually designed, while the neural architecturesearch networks are obtained using network architecture search technology based on the four types of architectures. Thesefive categories of real-time semantic segmentation methods cover almost all real-time semantic segmentation algorithmsbased on deep learning, including CNN-based, Transformer-based, and hybrid-architecture-based segmentation networks.Moreover, commonly used datasets and evaluation indicators of accuracy, speed, and model size are introduced for realtimesegmentation. We divided popular datasets into the autonomous driving scene and general scene datasets , and theevaluation indicators are divided into accuracy indicators and efficiency descriptors. In addition, this study quantitativelyevaluates various real-time semantic segmentation algorithms mentioned on multiple datasets by using relevantevaluation indicators. To avoid the interference of different devices when conducting a quantitative comparison betweenreal-time semantic segmentation algorithms, this study compares the performance of advanced methods of each categorywith the same devices and configuration and establishes a fair and integral real-time semantic segmentation evaluationsystem for subsequent research, thereby contributing to a unified standard for comparison. Finally, current challengesin real-time semantic segmentation are discussed, and possible future directions for improvements are envisioned(e. g. , utilization of Transformers , applications on edge devices, knowledge transfer of visual foundation models, diversityof evaluation indicators , variety of datasets , utilization of multimodal data and weakly supervised methods , combinationwith incremental learning ). The algorithms , datasets , and evaluation indicators mentioned in this paper are summarizedat https://github. com/xzz777/Awesome-Real-time-Semantic-Segmentation for the convenience of subsequentresearchers.