Multi-Stream Temporally Enhanced Network for Video Salient Object Detection

Video salient object detection(VSOD)aims at locating the most attractive objects in a video by exploring the spatial and temporal features.VSOD poses a challenging task in computer vision,as it involves processing complex spatial data that is also influenced by temporal dynamics.Despite the progress made in existing VSOD models,they still struggle in scenes of great background diversity within and between frames.Additionally,they encounter difficulties related to accumulated noise and high time consumption during the extraction of temporal features over a long-term duration.We propose a multi-stream temporal enhanced network(MSTENet)to address these problems.It investigates saliency cues collaboration in the spatial domain with a multi-stream structure to deal with the great background diversity challenge.A straightforward,yet efficient approach for temporal feature extraction is developed to avoid the accumulative noises and reduce time consumption.The distinction between MSTENet and other VSOD methods stems from its incorporation of both foreground supervision and background supervision,facilitating enhanced extraction of collaborative saliency cues.Another notable differentiation is the innovative integration of spatial and temporal features,wherein the temporal module is integrated into the multi-stream structure,enabling comprehensive spatial-temporal interactions within an end-to-end framework.Extensive experimental results demonstrate that the proposed method achieves state-of-the-art performance on five benchmark datasets while maintaining a real-time speed of 27 fps(Titan XP).Our code and models are available at https://github.com/RuJiaLe/MSTENet.

Local saliency consistency-based label inference for weakly supervised salient object detection using scribble annotations

Recently,weak supervision has received growing attention in the field of salient object detection due to the convenience of labelling.However,there is a large performance gap between weakly supervised and fully supervised salient object detectors because the scribble annotation can only provide very limited foreground/background information.Therefore,an intuitive idea is to infer annotations that cover more complete object and background regions for training.To this end,a label inference strategy is proposed based on the assumption that pixels with similar colours and close positions should have consistent labels.Specifically,k-means clustering algorithm was first performed on both colours and coordinates of original annotations,and then assigned the same labels to points having similar colours with colour cluster centres and near coordinate cluster centres.Next,the same annotations for pixels with similar colours within each kernel neighbourhood was set further.Extensive experiments on six benchmarks demonstrate that our method can significantly improve the performance and achieve the state-of-the-art results.

Salient Object Detection from Multi-spectral Remote Sensing Images with Deep Residual Network

alient object detection aims at identifying the visually interesting object regions that are consistent with human perception. Multispectral remote sensing images provide rich radiometric information in revealing the physical properties of the observed objects, which leads to great potential to perform salient object detection for remote sensing images. Conventional salient object detection methods often employ handcrafted features to predict saliency by evaluating the pixel-wise or superpixel-wise contrast. With the recent use of deep learning framework, in particular, fully convolutional neural networks, there has been profound progress in visual saliency detection. However, this success has not been extended to multispectral remote sensing images, and existing multispectral salient object detection methods are still mainly based on handcrafted features, essentially due to the difficulties in image acquisition and labeling. In this paper, we propose a novel deep residual network based on a top-down model, which is trained in an end-to-end manner to tackle the above issues in multispectral salient object detection. Our model effectively exploits the saliency cues at different levels of the deep residual network. To overcome the limited availability of remote sensing images in training of our deep residual network, we also introduce a new spectral image reconstruction model that can generate multispectral images from RGB images. Our extensive experimental results using both multispectral and RGB salient object detection datasets demonstrate a significant performance improvement of more than 10% improvement compared with the state-of-the-art methods.

Salient Object Detection Based on a Novel Combination Framework Using the Perceptual Matching and Subjective-Objective Mapping Technologies

The integrity and fineness characterization of non-connected regions and contours is a major challenge for existing salient object detection.The key to address is how to make full use of the subjective and objective structural information obtained in different steps.Therefore,by simulating the human visual mechanism,this paper proposes a novel multi-decoder matching correction network and subjective structural loss.Specifically,the loss pays different attentions to the foreground,boundary,and background of ground truth map in a top-down structure.And the perceived saliency is mapped to the corresponding objective structure of the prediction map,which is extracted in a bottom-up manner.Thus,multi-level salient features can be effectively detected with the loss as constraint.And then,through the mapping of improved binary cross entropy loss,the differences between salient regions and objects are checked to pay attention to the error prone region to achieve excellent error sensitivity.Finally,through tracking the identifying feature horizontally and vertically,the subjective and objective interaction is maximized.Extensive experiments on five benchmark datasets demonstrate that compared with 12 state-of-the-art methods,the algorithm has higher recall and precision,less error and strong robustness and generalization ability,and can predict complete and refined saliency maps.

RGB-T显著性目标检测综述

除RGB图像外,热红外图像也能提取出对显著性目标检测至关重要的显著性信息。热红外图像随着红外传感设备的发展和普及已经变得易于获取,RGB-T显著性目标检测已成为了热门研究领域,但目前仍缺少对现有方法全面的综述。首先介绍了基于机器学习的RGB-T显著性目标检测方法,然后着重介绍了两类基于深度学习的RGB-T显著性目标检测方法:基于卷积神经网络和基于Vision Transformer的方法。随后对相关数据集和评价指标进行介绍,并在这些数据集上对代表性的方法进行了定性和定量的比较分析。最后对RGB-T显著性目标检测面临的挑战及未来的发展方向进行了总结与展望。

A Novel Divide and Conquer Solution for Long-term Video Salient Object Detection

Recently,a new research trend in our video salient object detection(VSOD)research community has focused on enhancing the detection results via model self-fine-tuning using sparsely mined high-quality keyframes from the given sequence.Although such a learning scheme is generally effective,it has a critical limitation,i.e.,the model learned on sparse frames only possesses weak generalization ability.This situation could become worse on“long”videos since they tend to have intensive scene variations.Moreover,in such videos,the keyframe information from a longer time span is less relevant to the previous,which could also cause learning conflict and deteriorate the model performance.Thus,the learning scheme is usually incapable of handling complex pattern modeling.To solve this problem,we propose a divide-and-conquer framework,which can convert a complex problem domain into multiple simple ones.First,we devise a novel background consistency analysis(BCA)which effectively divides the mined frames into disjoint groups.Then for each group,we assign an individual deep model on it to capture its key attribute during the fine-tuning phase.During the testing phase,we design a model-matching strategy,which could dynamically select the best-matched model from those fine-tuned ones to handle the given testing frame.Comprehensive experiments show that our method can adapt severe background appearance variation coupling with object movement and obtain robust saliency detection compared with the previous scheme and the state-of-the-art methods.

Multi-Object Tracking with Micro Aerial Vehicle

A simple yet efficient tracking framework is proposed for real-time multi-object tracking with micro aerial vehicles(MAVs). It's basic missions for MAVs to detect specific targets and then track them automatically. In our method, candidate regions are generated using the salient detection in each frame and then classified by an eural network. A kernelized correlation filter(KCF) is employed to track each target until it disappears or the peak-sidelobe ratio is lower than a threshold. Besides, we define the birth and death of each tracker for the targets. The tracker is recycled if its target disappears and can be assigned to a new target. The algorithm is evaluated on the PAFISS and UAV123 datasets. The results show a good performance on both the tracking accuracy and speed.

边缘信息增强的显著性目标检测网络

针对显著性目标检测任务中识别结果边缘模糊的问题,提出了一种能够充分利用边缘信息增强边缘像素置信度的新模型。该网络主要有两个创新点:设计三重注意力模块,利用预测图的特点直接生成前景、背景和边缘注意力,并且生成注意力权重的过程不增加任何参数;设计边缘预测模块,在分辨率较高的网络浅层进行有监督的边缘预测,并与网络深层的显著图预测融合,细化了边缘。在6种常用公开数据集上用定性和定量的方法评估了该模型,并且与其他模型进行充分对比,证明设计的新模型能够取得最优的效果。此外,该模型参数量为30.28 M,可以在GTX 1080 Ti显卡上达到31帧·s^(-1)的预测速度。

基于特征注意力提纯的显著性目标检测模型

近年来,显著性目标检测技术取得了巨大进展,其中如何选择并有效集成多尺度特征扮演了重要角色。针对现有特征集成方法可能导致的信息冗余问题,提出了一种基于特征注意力提纯的显著性检测模型。首先,在解码器中采用一个全局特征注意力引导模块(GAGM)对带有语义信息的深层特征进行注意力机制处理,得到全局上下文信息;然后,通过全局引导流将其送入解码器各层进行监督训练;最后,利用多尺度特征融合模块(FAM)对编码器提取出的多尺度特征与全局上下文信息进行有效集成,并在网格状特征提纯模块(MFPM)中进行进一步细化,以生成清晰、完整的显著图。在5个公开数据集上进行实验,结果表明,所提模型优于现有的其他显著性检测方法,并且处理速度快,当处理320×320尺寸的图像时,能以30帧以上的速度运行。

面向360度全景图像显著目标检测的相邻协调网络

为解决360°全景图像显著目标检测(SOD)中的显著目标尺度变化和边缘不连续、易模糊的问题,该文提出一种基于相邻协调网络的360°全景图像显著目标检测方法(ACoNet)。首先,利用相邻细节融合模块获取相邻特征中的细节和边缘信息,以促进显著目标的精确定位。其次,使用语义引导特征聚合模块来聚合浅层特征和深层特征之间不同尺度上的语义特征信息,并抑制浅层特征传递的噪声,缓解解码阶段显著目标与背景区域不连续、边界易模糊的问题。同时构建多尺度语义融合子模块扩大不同卷积层的多尺度感受野,实现精确训练显著目标边界的效果。在2个公开的数据集上进行的大量实验结果表明,相比于其他13种先进方法,所提方法在6个客观评价指标上均有明显的提升,同时主观可视化检测的显著图边缘轮廓性更好,空间结构细节信息更清晰。

基于先验特征聚类的目标检测优化方法

针对显著目标检测问题在没有任何先验信息的情况下,通过特征聚类和紧致性先验方案实现目标检测优化。优化后的方法包括四个步骤:首先采用超像素预处理将图像分割成超像素,以抑制噪声并降低计算复杂度;其次应用改进的虾群聚类算法对颜色特征进行分类;接着利用二维熵来衡量每个簇的紧密度,并构建背景模型;最后以背景区域与其他区域之间的对比度作为显著特征,并通过设计高斯滤波器增强其显著性。为了更好地评价显著目标检测的精度,本文通过多维评价指标进行优劣性实验分析,实验结果表明,文中算法具有较好的实时性与鲁棒性。

集成多种上下文与混合交互的显著性目标检测

显著性目标检测目的是识别和分割图像中的视觉显著性目标,它是计算机视觉任务及其相关领域的重要研究内容之一。当下基于全卷积网络(FCNs)的显著性目标检测方法已经取得了不错的性能,然而现实场景中的显著性目标类型多变且尺寸不固定,这使得准确检测并完整分割出显著性目标仍然是一个巨大的挑战。为此,该文提出集成多种上下文和混合交互的显著性目标检测方法,通过利用密集上下文信息探索模块和多源特征混合交互模块来高效预测显著性目标。密集上下文信息探索模块采用空洞卷积、不对称卷积和密集引导连接渐进地捕获具有强关联性的多尺度和多感受野上下文信息,通过集成这些信息来增强每个初始多层级特征的表达能力。多源特征混合交互模块包含多种特征聚合操作,可以自适应交互来自多层级特征中的互补性信息,以生成用于准确预测显著性图的高质量特征表示。此方法在5个公共数据集上进行了性能测试,实验结果表明,该文方法在不同的评估指标下与19种基于深度学习的显著性目标检测方法相比取得优越的预测性能。

基于多尺度视觉感知特征融合的显著目标检测方法

显著性物体检测具有重要的理论研究意义和实际应用价值,已在许多计算机视觉应用中发挥了重要作用,如视觉追踪、图像分割、物体识别等。然而,自然环境下显著目标的类别未知、尺度多变依然是物体检测面临的一大挑战,影响着显著目标的检测效果。因此,提出了一种基于多尺度视觉感知特征融合的显著目标检测方法。首先,基于视觉感知显著目标的特性,设计并提取多个图像感知特征。其次,图像感知特征采用多尺度自适应方式,获取特征显著图。然后,将各个显著特征图融合,获得最终的显著目标。该方法基于不同图像感知特征的特点,自适应提取显著目标,能够适应多变的检测目标与复杂的检测环境。实验结果表明,在受自然环境中背景干扰的情况下,该方法能有效检测出未知类别和不同尺度的显著目标。

SAM Era:Can It Segment Any Industrial Surface Defects?

Segment Anything Model(SAM)is a cutting-edge model that has shown impressive performance in general object segmentation.The birth of the segment anything is a groundbreaking step towards creating a universal intelligent model.Due to its superior performance in general object segmentation,it quickly gained attention and interest.This makes SAM particularly attractive in industrial surface defect segmentation,especially for complex industrial scenes with limited training data.However,its segmentation ability for specific industrial scenes remains unknown.Therefore,in this work,we select three representative and complex industrial surface defect detection scenarios,namely strip steel surface defects,tile surface defects,and rail surface defects,to evaluate the segmentation performance of SAM.Our results show that although SAM has great potential in general object segmentation,it cannot achieve satisfactory performance in complex industrial scenes.Our test results are available at:https://github.com/VDT-2048/SAM-IS.

视觉显著物体检测综述:方法、挑战及未来

视觉显著物体检测作为计算机视觉领域的关键研究方向,也是学术研究的热点之一。本文系统性地梳理了该领域的研究方法、面临的挑战和未来的发展方向。首先,概述了视觉显著物体检测的发展脉络,以及其在计算机视觉领域的广泛应用;其次,对视觉显著物体检测方法进行了详尽的回顾,涵盖了基于显著性特征以及深度学习框架下的检测方法;再次,深入探讨了基于传统卷积神经网络和全卷积神经网络的显著物体检测方法,以及基于注意力机制的显著物体检测方法,并对视觉显著物体检测领域常用的数据集和评价指标进行了介绍;从次,针对当前视觉显著物体检测面临的挑战,如现有数据集的局限、复杂场景下的检测准确度等,文章进行了总结分析;最后,展望了视觉显著物体检测的未来发展方向。通过本文的综述,旨在为从事视觉显著物体检测的研究者提供全面而深入的参考,以促进该领域的进一步发展。

跨模态交互融合与全局感知的RGB-D显著性目标检测

近年来,RGB-D显著性检测方法凭借深度图中丰富的几何结构和空间位置信息,取得了比RGB显著性检测模型更好的性能,受到学术界的高度关注.然而,现有的RGB-D检测模型仍面临着持续提升检测性能的需求.最近兴起的Transformer擅长建模全局信息,而卷积神经网络(CNN)擅长提取局部细节.因此,如何有效结合CNN和Transformer两者的优势,挖掘全局和局部信息,将有助于提升显著性目标检测的精度.为此,提出一种基于跨模态交互融合与全局感知的RGB-D显著性目标检测方法,通过将Transformer网络嵌入U-Net中,从而将全局注意力机制与局部卷积结合在一起,能够更好地对特征进行提取.首先借助U-Net编码-解码结构,高效地提取多层次互补特征并逐级解码生成显著特征图.然后,使用Transformer模块学习高级特征间的全局依赖关系增强特征表示,并针对输入采用渐进上采样融合策略以减少噪声信息的引入.其次,为了减轻低质量深度图带来的负面影响,设计一个跨模态交互融合模块以实现跨模态特征融合.最后,5个基准数据集上的实验结果表明,所提算法与其他最新的算法相比具有显著优势.

基于特征重聚焦和精细化的遥感显著性目标检测

为了提升网络对特征的表征,提出一种基于特征重聚焦和精细化的光学遥感显著目标检测算法。利用相邻层特征交互捕获上下文语义互补信息,并通过膨胀卷积调节感受野提取信息的范围,完成初次特征聚焦。再将注意机制作用于深层特征,组成位置引导模块,增强对显著性特征的关注,完成特征重聚焦。最后,通过浅层特征获得显著特征注意图和反注意图,引导网络进一步挖掘高置信度显著区域和低置信度背景区域的信息,精细化优化后的特征。采用EORSSD和ORSSD 2个公开数据集进行实验与评估,以证明算法的有效性。

基于多任务学习的视频和图像显著目标检测方法

显著目标检测(Salient Object Detection,SOD)能够模拟人类的注意力机制,在复杂的场景中快速发现高价值的显著目标,为进一步的视觉理解任务奠定了基础。当前主流的图像显著目标检测方法通常基于DUTS-TR数据集进行训练,而视频显著目标检测方法(Video Salient Object Detection,VSOD)基于DAVIS,DAVSOD以及DUTS-TR数据集进行训练。图像和视频显著目标检测任务既有共性又有特性,因此需要部署独立的模型进行单独训练,这大大增加了运算资源和训练时间的开销。当前研究大多针对单个任务提出独立的解决方案,而缺少统一的图像和视频显著目标检测方法。针对上述问题,提出了一种基于多任务学习的图像和视频显著目标检测方法,旨在构建一种通用的模型框架,通过一次训练同时适配两种任务,并进一步弥合图像和视频显著目标检测方法之间的性能差异。12个数据集上的定性和定量实验结果表明,所提方法不仅能够同时适配两种任务,而且取得了比单任务模型更好的检测结果。

基于伪标签的弱监督显著特征增强目标检测方法

显著性目标检测旨在检测图像中最明显的区域。传统的基于单一标签的算法不可避免地受到所采用的细化算法的影响,表现出偏见特征,从而进一步影响了显著性网络的检测性能。针对这一问题,基于多指令滤波器结构,提出了一种基于伪标签的弱监督显著特征增强目标检测方法FeaEM,通过从多个标签中集成更全面和准确的显著性线索,从而有效提升目标检测的性能。FeaEM方法的核心是引入一个新的多指令滤波器结构,利用多个伪标签来避免单一标签带来的负面影响;通过在指令滤波器中引入特征选择机制,从噪声伪标签中提取和过滤更准确的显著性线索,从而学习更多有效的具有代表性的特征;同时,针对现有的弱监督目标检测方法对输入图像的尺度十分敏感,同一图像的不同尺寸输入的预测结构存在较大偏差问题,通过引入尺度特征融合机制,以确保在输入不同尺寸的同一图像时,能输出一致的显著图,进而有效提高模型的尺度泛化能力。在多个数据集上进行的大量实验表明,所提出的FeaEM方法优于最具代表性的方法。

Salient object detection: A survey

Detecting and segmenting salient objects from natural scenes, often referred to as salient object detection, has attracted great interest in computer vision. While many models have been proposed and several applications have emerged, a deep understanding of achievements and issues remains lacking. We aim to provide a comprehensive review of recent progress in salient object detection and situate this field among other closely related areas such as generic scene segmentation, object proposal generation, and saliency for fixation prediction. Covering 228 publications, we survey i) roots, key concepts, and tasks, ii) core techniques and main modeling trends, and iii) datasets and evaluation metrics for salient object detection. We also discuss open problems such as evaluation metrics and dataset bias in model performance, and suggest future research directions.