SMSTracker:A Self-Calibration Multi-Head Self-Attention Transformer for Visual Object Tracking

Visual object tracking plays a crucial role in computer vision.In recent years,researchers have proposed various methods to achieve high-performance object tracking.Among these,methods based on Transformers have become a research hotspot due to their ability to globally model and contextualize information.However,current Transformer-based object tracking methods still face challenges such as low tracking accuracy and the presence of redundant feature information.In this paper,we introduce self-calibration multi-head self-attention Transformer(SMSTracker)as a solution to these challenges.It employs a hybrid tensor decomposition self-organizing multihead self-attention transformermechanism,which not only compresses and accelerates Transformer operations but also significantly reduces redundant data,thereby enhancing the accuracy and efficiency of tracking.Additionally,we introduce a self-calibration attention fusion block to resolve common issues of attention ambiguities and inconsistencies found in traditional trackingmethods,ensuring the stability and reliability of tracking performance across various scenarios.By integrating a hybrid tensor decomposition approach with a self-organizingmulti-head self-attentive transformer mechanism,SMSTracker enhances the efficiency and accuracy of the tracking process.Experimental results show that SMSTracker achieves competitive performance in visual object tracking,promising more robust and efficient tracking systems,demonstrating its potential to providemore robust and efficient tracking solutions in real-world applications.

基于Depth-wise卷积和视觉Transformer的图像分类模型

图像分类作为一种常见的视觉识别任务,有着广阔的应用场景。在处理图像分类问题时,传统的方法通常使用卷积神经网络,然而,卷积网络的感受野有限,难以建模图像的全局关系表示,导致分类精度低,难以处理复杂多样的图像数据。为了对全局关系进行建模,一些研究者将Transformer应用于图像分类任务,但为了满足Transformer的序列化和并行化要求,需要将图像分割成大小相等、互不重叠的图像块,破坏了相邻图像数据块之间的局部信息。此外,由于Transformer具有较少的先验知识,模型往往需要在大规模数据集上进行预训练,因此计算复杂度较高。为了同时建模图像相邻块之间的局部信息并充分利用图像的全局信息,提出了一种基于Depth-wise卷积的视觉Transformer(Efficient Pyramid Vision Transformer,EPVT)模型。EPVT模型可以实现以较低的计算成本提取相邻图像块之间的局部和全局信息。EPVT模型主要包含3个关键组件:局部感知模块(Local Perceptron Module,LPM)、空间信息融合模块(Spatial Information Fusion,SIF)和“+卷积前馈神经网络(Convolution Feed-forward Network,CFFN)。LPM模块用于捕获图像的局部相关性;SIF模块用于融合相邻图像块之间的局部信息,并利用不同图像块之间的远距离依赖关系,提升模型的特征表达能力,使模型学习到输出特征在不同维度下的语义信息;CFFN模块用于编码位置信息和重塑张量。在图像分类数据集ImageNet-1K上,所提模型优于现有的同等规模的视觉Transformer分类模型,取得了82.6%的分类准确度,证明了该模型在大规模数据集上具有竞争力。

基于Transformer视觉特征融合的图像描述方法

现有图像描述方法只利用区域型视觉特征生成描述语句,忽略了网格型视觉特征的重要性,并且均为两阶段方法,从而影响了图像描述的质量。针对该问题,提出一种基于Transformer视觉特征融合的端到端图像描述方法。首先,在特征提取阶段,利用视觉特征提取器提取出区域型视觉特征和网格型视觉特征;其次,在特征融合阶段,通过视觉特征融合模块对区域型视觉特征和网格型视觉特征进行拼接;最后,将所有的视觉特征送入语言生成器中以生成图像描述。该方法各部分均基于Transformer模型实现,实现了一阶段方法。在MS-COCO数据集上的实验结果表明,所提方法能够充分利用区域型视觉特征与网格型视觉特征的优势,BLEU-1、BLEU-4、METEOR、ROUGE-L、CIDEr、SPICE指标分别达到83.1%、41.5%、30.2%、60.1%、140.3%、23.9%,优于目前主流的图像描述方法,能够生成更加准确和丰富的描述语句。

基于多层级视频Transformer的视觉自动定位方法

工业自动化产线中,设备的异常检测直接决定加工质量,由机械臂和搭载于机械臂前端的工业相机构成的视觉系统可以有效监测此类异常。本文使用六轴机械臂搭载工业相机对工件表面进行成像,获取由模糊到清晰再到模糊的视频序列,以此选出最清晰的视频帧作为自动加工中有聚焦要求的距离指导,以进行聚焦异常修正,从而实现自动定位。提出一种基于多层级视频Transformer的视频分类模型多级视频Transformer(MLVT)用于高语义级别的视频表征学习,并用于选出视频序列中成像最清晰的帧。首先,提出一种具有多种感受野的token划分方法多级标记(MLT),能够将原始视频数据按2D图像补丁、3D图像补丁、帧和片段这4个层级划分成token序列,并在加入位置编码之后送入多级编码器(MLE)方法进行注意力的计算。为了缓解多层级的tokens带来的计算代价和收敛速度慢的问题,MLE引入一种逐层的可变形注意力机制逐层可变形注意力机制(LWLA),以一种可学习的方式代替全局注意力进行特征相似性的计算。最终,该方法3个版本的模型在本文的视频数据集上分别取得了87.2%、88.6%、88.9%的分类准确率,在与同参数量级的主流视频Transformer实验对比中均表现了最优的性能,有效地完成了从视频序列中选择出最清晰帧的任务,能够为下游视觉任务的性能提供强有力保障。

结合视觉Transformer和CNN的道路裂缝检测方法

提出了一种结合视觉Transformer和CNN的道路裂缝检测方法。利用CNN来捕获局部的细节信息,同时利用视觉Transformer来捕获全局特征。通过设计的Fusion特征融合模块将两者提取的特征有机地结合在一起,从而解决了单独使用CNN或视觉Transformer方法存在的局限。最终将结果传递至交互式解码器,生成道路裂缝的检测结果。实验结果表明,无论是在公开的数据集上还是在自建的数据集上,相较于单独使用CNN或视觉Transformer的方法,所提出的方法在道路裂缝检测任务中有更好的效果。

时域孪生网络融合Transformer的长时无人机视觉跟踪

针对无人机(UAV)执行跟踪任务时经常出现尺寸变化、低分辨率、目标遮挡等场景导致跟踪目标框漂移的问题,提出一种时域孪生网络融合Transformer的长时无人机视觉跟踪算法TTTrack。首先,使用基于孪生网络的SiamFC++(AlexNet)算法作为基线算法;其次,利用Transformer自适应地提取历史帧的时空信息并在线更新模板,从而将时空上下文信息储存为动态模板;随后,分别使用基准模板和动态模板与搜索特征图进行互相关运算,获得响应图后利用Transformer融合两个响应图,从而在连续帧之间建立时空上下文映射关系。实验结果表明,在LaSOT长序列跟踪基准上TTTrack的成功率和精确率分别为63.9%和66.6%,在UAV123跟踪基准上的成功率和精确率分别为61.4%和80.2%。与基线算法相比,该算法在完全遮挡场景下的成功率和精确率分别提升7.4和8.0个百分点。TTTrack在DTB70跟踪基准上精确率达到82.1%,并且跟踪速度为118 帧/s,满足实时性要求。测试结果验证了TTTrack具有良好的鲁棒性、实时性和抗干扰能力,能有效应对长时UAV跟踪任务。

数字赋能:基于视觉Transformer的非遗苗绣纹样数字化提取

苗绣通常是以线稿描绘的基础图样,搭配丰富色彩的丝线及不同的刺绣技法而形成,出现在一系列如服饰、头饰及配件等纺织品中,但是随着纺织品的损坏和丢失等原因,部分苗绣纹样便也随之流失。目前采用传统手工描绘纹样获取线稿的方式极其不便,因此文章针对苗绣纹样的数字化提取,提出了一种基于两阶段渐进采样视觉Transformer的边缘检测算法,分为全局和局部检测。在两个阶段都引入渐进式采样来定位重要区域,使提取的边缘集中于苗绣纹样主体部分,减少服饰背景等造成的干扰。通过使用多尺度通道注意力特征融合模块,将全局和局部检测的边缘进行加权融合,以获得更清晰的边缘。实验结果表明,该算法与其他算法相比,提取的苗绣纹样获得了更纤细的线条,且丢失的纹样形状线条较少,纹样整体效果与标签图最接近,效果最佳。

集成卷积神经网络和视觉Transformer的隧道掌子面岩性判识研究

为研究综合高效的隧道掌子面岩性智能分类方法,首先,通过收集高铁沿线施工隧道高清掌子面照片、地质素描图及工程地质说明,筛选并统计出灰岩、泥岩、砂岩、玄武岩4种岩性,在此基础上,采用图像增强扩充样本数量并构建岩性样本集;然后,基于上述样本集分别构建ResNet50V2岩性分类迁移模型及VIT岩性分类模型,对比二者岩性分类效果,并采用Stacking方法集成2种模型的分类特点;最后,通过对比3种元学习器(逻辑回归、支持向量机、决策树)对2种模型的集成融合效果来选取最适用的元学习器。结果表明:采用逻辑回归集成ResNet50V2及VIT所构建的集成模型对岩性的分类效果最好,能充分融合掌子面岩性的全、局部特征来进行分类,模型准确率达到93.8%。

MSD-Net: Pneumonia Classification Model Based on Multi-Scale Directional Feature Enhancement

Computer-aided diagnosis of pneumonia based on deep learning is a research hotspot.However,there are some problems that the features of different sizes and different directions are not sufficient when extracting the features in lung X-ray images.A pneumonia classification model based on multi-scale directional feature enhancement MSD-Net is proposed in this paper.The main innovations are as follows:Firstly,the Multi-scale Residual Feature Extraction Module(MRFEM)is designed to effectively extract multi-scale features.The MRFEM uses dilated convolutions with different expansion rates to increase the receptive field and extract multi-scale features effectively.Secondly,the Multi-scale Directional Feature Perception Module(MDFPM)is designed,which uses a three-branch structure of different sizes convolution to transmit direction feature layer by layer,and focuses on the target region to enhance the feature information.Thirdly,the Axial Compression Former Module(ACFM)is designed to perform global calculations to enhance the perception ability of global features in different directions.To verify the effectiveness of the MSD-Net,comparative experiments and ablation experiments are carried out.In the COVID-19 RADIOGRAPHY DATABASE,the Accuracy,Recall,Precision,F1 Score,and Specificity of MSD-Net are 97.76%,95.57%,95.52%,95.52%,and 98.51%,respectively.In the chest X-ray dataset,the Accuracy,Recall,Precision,F1 Score and Specificity of MSD-Net are 97.78%,95.22%,96.49%,95.58%,and 98.11%,respectively.This model improves the accuracy of lung image recognition effectively and provides an important clinical reference to pneumonia Computer-Aided Diagnosis.

Multiscale Fusion Transformer Network for Hyperspectral Image Classification

Convolutional neural network(CNN)has excellent ability to model locally contextual information.However,CNNs face challenges for descripting long-range semantic features,which will lead to relatively low classification accuracy of hyperspectral images.To address this problem,this article proposes an algorithm based on multiscale fusion and transformer network for hyperspectral image classification.Firstly,the low-level spatial-spectral features are extracted by multi-scale residual structure.Secondly,an attention module is introduced to focus on the more important spatialspectral information.Finally,high-level semantic features are represented and learned by a token learner and an improved transformer encoder.The proposed algorithm is compared with six classical hyperspectral classification algorithms on real hyperspectral images.The experimental results show that the proposed algorithm effectively improves the land cover classification accuracy of hyperspectral images.

基于CNN-Transformer的自动泊车车位感知算法

为提高自动泊车成功率及准确性,首先基于卷积神经网络(CNN)模型对输入图像进行特征提取,然后利用Transfomer模型的“编码-解码”机制对CNN提取到的图像特征平铺后进行计算推理,通过前馈神经网络得到目标预测结果,最后基于180°广角鱼眼图像进行推理识别,车位角中心点和空车位入口中心点均采用二维坐标表示,降低了输出信息的冗余,优化了模型结构。测试结果表明,该算法能够较好地适应不同车位线划线方式和不同的自然环境,目标感知的召回率达到98%,车位角中心点定位平均误差小于3 cm,满足泊车系统对车位感知的鲁棒性、实时性和准确性要求。

Integrating Transformer and Bidirectional Long Short-Term Memory for Intelligent Breast Cancer Detection from Histopathology Biopsy Images

Breast cancer is a significant threat to the global population,affecting not only women but also a threat to the entire population.With recent advancements in digital pathology,Eosin and hematoxylin images provide enhanced clarity in examiningmicroscopic features of breast tissues based on their staining properties.Early cancer detection facilitates the quickening of the therapeutic process,thereby increasing survival rates.The analysis made by medical professionals,especially pathologists,is time-consuming and challenging,and there arises a need for automated breast cancer detection systems.The upcoming artificial intelligence platforms,especially deep learning models,play an important role in image diagnosis and prediction.Initially,the histopathology biopsy images are taken from standard data sources.Further,the gathered images are given as input to the Multi-Scale Dilated Vision Transformer,where the essential features are acquired.Subsequently,the features are subjected to the Bidirectional Long Short-Term Memory(Bi-LSTM)for classifying the breast cancer disorder.The efficacy of the model is evaluated using divergent metrics.When compared with other methods,the proposed work reveals that it offers impressive results for detection.

基于CNN和Transformer混合网络模型的车道线检测

车道线检测技术在自动驾驶系统中发挥着重要作用,目前基于深度学习的车道线检测方法通常在主干网络提取特征之后分别获取车道线关键点的置信度以及这些点相对车道线起始点的偏移。但由于车道线是细长结构,现有的主干网络无法有效提取这种结构特征,偏移网络也难以回归车道线上关键点相对起始点的偏移。鉴于注意力机制在提取空间结构特征、表征长距离图像序列间依赖关系方面的优越性能,在基于点的车道线检测方法的基础上提出了一种基于卷积神经网络(convolutional neural network,CNN)和Transformer的混合网络(CNN-Transformer hybrid network,CTNet)模型,该模型通过特征金字塔和增强的坐标注意力机制提高特征的表征能力,使用基于视觉Transformer的偏移网络回归关键点的偏移量,因此,CTNet能够提取细长车道线特征、捕获长距离点间的偏移,有效提升车道线检测的精度。实验对比了CTNet和6种常用车道线检测算法在数据集TuSimple和CULane上的效果,在TuSimple上CTNet各项精度指标均优于现有方法,在CULane数据集的9种不同车道场景中,CTNet在6个场景中取得了最佳精度。

基于双循环Transformer的三维人体姿态估计

针对视觉传感器采集到的图像进行三维人体姿态估计,提出一种双循环Transformer网络模型,有效地从二维关键关节点中提取时空维度高相关性特征,增大感受野,从而提高三维姿态估计的精度。通过在视觉传感器采集得到的公开数据集Human3.6M上的仿真实验,验证了双循环Transformer算法的性能。分析结果表明,最终估计得到的三维人体关节点的平均关节点位置偏差MPJPE(Mean Per Joint Position Error)为41.6 mm,相比于现有方法有一定提升,可以应用到许多下游相关工作中,有着较强的应用价值。

改进视觉Transformer的人脸识别方法

目前大多数人脸识别方法依赖于卷积神经网络,通过级联的形式构建多层处理单元,利用卷积操作融合局部特征,忽略了人脸全局语义信息,缺乏对人脸重点区域的关注度。针对上述问题,提出一种基于改进视觉Transformer的人脸识别方法,引入Shuffle Transformer作为特征提取骨干网络,通过自注意力机制以及Shuffle操作捕捉特征图全局信息,建立特征点之间的长距离依赖关系,提高模型的特征感知能力;同时,结合ArcFace损失函数和中心损失函数的特点,设计融合损失作为目标函数,利用类内约束扩大角度间隔,提高特征空间的辨别性。该方法在LFW、CALFW、CPLFW、AgeDB-30和CFP五个具有挑战性的基准测试人脸数据集上分别取得了99.83%、95.87%、90.05%、98.05%、97.23%的平均准确率,能够有效提升人脸特征提取能力,识别效果优于同等规模卷积神经网络。

基于自注意力Transformer编码器的多阶段电力系统暂态稳定评估方法

人工智能方法在电力系统暂态稳定评估研究中已经取得了一定的成果。常规深层网络普遍被视为“黑盒”模型,这限制了智能算法在实际工程应用中的可信赖性;同时,常规算法对电力系统时序信息的提取能力不足。针对以上问题,构建基于Transformer编码器的多阶段暂态稳定评估方法,其多阶段预测能够有效降低失稳漏判率。和常规算法相比,Transformer模型具有良好的可解释性,其注意力机制引导模型自适应识别并聚焦于关键特征,在一定程度上揭示深层网络内部工作决策过程。此外,采用多时刻信息构建特征空间,Transformer通过注意力机制实现全局感受野,使模型快速捕获电力系统前后时刻间的状态依赖。IEEE-39节点系统上的仿真结果表明,所提方法相比常见数据驱动模型具有更高的暂稳评估准确性,呈现出良好的可解释性,并在数据污染时依然维持较高的性能。

基于辅助任务和Transformer的人脸正面化网络

现有人脸正面化方法仅使用侧面图像生成正面图像,容易带来生成效果不佳及过拟合等问题。对此,提出一种具有辅助任务及Transformer的生成对抗网络(Auxiliary Task Generative Adversarial Network,AT-GAN)。AT-GAN利用多任务的相关性提高人脸正面化效果及泛化性,主任务为人脸正面化本身,使用侧面人脸生成对应的正面人脸;次任务为侧面人像草图生成对应的正面人像草图,引导并辅助主任务,加速网络收敛。两任务之间共享网络权重,并使用基于视觉Transformer的特征交互模块将两部分特征深度融合,提高网络整体的性能,生成更加具有真实感的正面图像。AT-GAN由生成器及判别器组成,生成器的特征提取部分将人脸关键点与空间注意力结合,确保模型准确地提取关键特征。实验结果表明,AT-GAN在MASFD与CAS-PEAL-R1数据集上的Rank-1识别率分别平均提高了4.42%与1.30%,视觉效果及模型泛化性得到提升。

Denoising of seismic data via multi-scale ridgelet transform

Noise has traditionally been suppressed or eliminated in seismic data sets by the use of Fourier filters and, to a lesser degree, nonlinear statistical filters. Although these methods are quite useful under specific conditions, they may produce undesirable effects for the low signal to noise ratio data. In this paper, a new method, multi-scale ridgelet transform, is used in the light of the theory of ridgelet transform. We employ wavelet transform to do sub-band decomposition for the signals and then use non-linear thresholding in ridgelet domain for every block. In other words, it is based on the idea of partition, at sufficiently fine scale, a curving singularity looks straight, and so ridgelet transform can work well in such cases. Applications on both synthetic data and actual seismic data from Sichuan basin, South China, show that the new method eliminates the noise portion of the signal more efficiently and retains a greater amount of geologic data than other methods, the quality and consecutiveness of seismic event are improved obviously as well as the quality of section is improved.

DT-Net:Joint Dual-Input Transformer and CNN for Retinal Vessel Segmentation

Retinal vessel segmentation in fundus images plays an essential role in the screening,diagnosis,and treatment of many diseases.The acquired fundus images generally have the following problems:uneven illumination,high noise,and complex structure.It makes vessel segmentation very challenging.Previous methods of retinal vascular segmentation mainly use convolutional neural networks on U Network(U-Net)models,and they have many limitations and shortcomings,such as the loss of microvascular details at the end of the vessels.We address the limitations of convolution by introducing the transformer into retinal vessel segmentation.Therefore,we propose a hybrid method for retinal vessel segmentation based on modulated deformable convolution and the transformer,named DT-Net.Firstly,multi-scale image features are extracted by deformable convolution and multi-head selfattention(MHSA).Secondly,image information is recovered,and vessel morphology is refined by the proposed transformer decoder block.Finally,the local prediction results are obtained by the side output layer.The accuracy of the vessel segmentation is improved by the hybrid loss function.Experimental results show that our method obtains good segmentation performance on Specificity(SP),Sensitivity(SE),Accuracy(ACC),Curve(AUC),and F1-score on three publicly available fundus datasets such as DRIVE,STARE,and CHASE_DB1.

Multi-scale Incremental Analysis Update Scheme and Its Application to Typhoon Mangkhut(2018)Prediction

In the traditional incremental analysis update(IAU)process,all analysis increments are treated as constant forcing in a model’s prognostic equations over a certain time window.This approach effectively reduces high-frequency oscillations introduced by data assimilation.However,as different scales of increments have unique evolutionary speeds and life histories in a numerical model,the traditional IAU scheme cannot fully meet the requirements of short-term forecasting for the damping of high-frequency noise and may even cause systematic drifts.Therefore,a multi-scale IAU scheme is proposed in this paper.Analysis increments were divided into different scale parts using a spatial filtering technique.For each scale increment,the optimal relaxation time in the IAU scheme was determined by the skill of the forecasting results.Finally,different scales of analysis increments were added to the model integration during their optimal relaxation time.The multi-scale IAU scheme can effectively reduce the noise and further improve the balance between large-scale and small-scale increments in the model initialization stage.To evaluate its performance,several numerical experiments were conducted to simulate the path and intensity of Typhoon Mangkhut(2018)and showed that:(1)the multi-scale IAU scheme had an obvious effect on noise control at the initial stage of data assimilation;(2)the optimal relaxation time for large-scale and small-scale increments was estimated as 6 h and 3 h,respectively;(3)the forecast performance of the multi-scale IAU scheme in the prediction of Typhoon Mangkhut(2018)was better than that of the traditional IAU scheme.The results demonstrate the superiority of the multi-scale IAU scheme.