小数据集上基于语义的局部注意视觉Transformer方法

在小数据集上从零开始训练时,视觉Transformer无法与同规模的卷积神经网络媲美。基于图像的局部注意力方法,可以显著提高ViT的数据效率,但是会丢失距离较远但相关的补丁之间的信息。为了解决上述问题,提出一种双向并行局部注意力视觉Transformer的方法。该方法首先在特征层面上对补丁进行分组,在组内执行局部注意力,以利用特征空间中补丁之间的关系弥补信息丢失。其次,为了有效融合补丁之间的信息,将基于语义的局部注意力和基于图像的局部注意力并行结合起来,通过双向自适应学习来增强ViT模型在小数据上的性能。实验结果表明,该方法在计算量为15.2 GFLOPs和参数量为57.2 M的情况下,分别在CIFAR-10和CIFAR-100数据集上实现了97.93%和85.80%的准确性。相比于其他方法,双向并行局部注意力视觉Transformer在增强局部引导能力的同时,保持了局部注意力所需属性的有效性。

基于Transformer模型的时序数据预测方法综述

时序数据预测(TSF)是指通过分析历史数据的趋势性、季节性等潜在信息,预测未来时间点或时间段的数值和趋势。时序数据由传感器生成,在金融、医疗、能源、交通、气象等众多领域都发挥着重要作用。随着物联网传感器的发展,海量的时序数据难以使用传统的机器学习解决,而Transformer在自然语言处理和计算机视觉等领域的诸多任务表现优秀,学者们利用Transformer模型有效捕获长期依赖关系,使得时序数据预测任务取得了飞速发展。综述了基于Transformer模型的时序数据预测方法,按时间梳理了时序数据预测的发展进程,系统介绍了时序数据预处理过程和方法,介绍了常用的时序预测评价指标和数据集。以算法框架为研究内容系统阐述了基于Transformer的各类模型在TSF任务中的应用方法和工作原理。通过实验对比了各个模型的性能、优点和局限性,并对实验结果展开了分析与讨论。结合Transformer模型在时序数据预测任务中现有工作存在的挑战提出了该方向未来发展趋势。

图像处理中CNN与视觉Transformer混合模型研究综述

卷积神经网络(CNN)与视觉Transformer是目前图像处理领域中两大重要的深度学习模型,两者经过多年来不断的研究与进步,已在该领域取得了非凡的成就。近些年来,CNN与视觉Transformer的混合模型正在逐步兴起,广泛的研究不断克服两种模型存在的弱项,高效地发挥出各自的亮点,在图像处理任务中表现出优异的效果。基于CNN与视觉Transformer混合模型进行深入阐述。总体概述了CNN与Vision Transformer模型的架构和优缺点,并总结混合模型的概念及优势。围绕串行结构融合方式、并行结构融合方式、层级交叉结构融合方式以及其他融合方式等四个方面全面回顾梳理了混合模型的研究现状和实际进展,并针对各种融合方式的主要代表模型进行总结与剖析,从多方面对典型混合模型进行评价对比。多角度叙述了混合模型在图像识别、图像分类、目标检测和图像分割等实际图像处理特定领域中应用研究,展现出混合模型在具体实践中的适用性和高效性。深入分析混合模型未来研究方向,并为后续该模型在图像处理中的研究与应用提出展望。

ViTAU:基于Vision transformer和面部动作单元的面瘫识别与分析

面部神经麻痹(Facial nerve paralysis,FNP),通常称为贝尔氏麻痹或面瘫,对患者的日常生活和心理健康产生显著影响,面瘫的及时识别和诊断对于患者的早期治疗和康复至关重要.随着深度学习和计算机视觉技术的快速发展,面瘫的自动识别变得可行,为诊断提供了一种更准确和客观的方式.目前的研究主要集中关注面部的整体变化,而忽略了面部细节的重要性.面部不同部位对识别结果的影响力并不相同,这些研究尚未对面部各个区域进行细致区分和分析.本项研究引入结合Vision transformer(ViT)模型和动作单元(Action unit,AU)区域检测网络的创新性方法用于面瘫的自动识别及区域分析.ViT模型通过自注意力机制精准识别是否面瘫,同时,基于AU的策略从StyleGAN2模型提取的特征图中,利用金字塔卷积神经网络分析受影响区域.这一综合方法在YouTube Facial Palsy(YFP)和经过扩展的Cohn Kanade(CK+)数据集上的实验中分别达到99.4%的面瘫识别准确率和81.36%的面瘫区域识别准确率.通过与最新方法的对比,实验结果展示了所提的自动面瘫识别方法的有效性.

基于贝叶斯图注意力Transformer的航空发动机剩余使用寿命概率预测

航空发动机作为飞机的心脏,其健康状态对飞机的安全飞行至关重要.深度学习强大的数据挖掘能力,为通过海量历史数据预测航空发动机的剩余使用寿命提供了新方法.然而,传统基于深度学习的方法大都关注于挖掘数据在时间上的关联,而忽略了多个传感器监测数据之间复杂的非欧氏空间关系.此外,少有研究考虑数据或者预测过程本身具有的不确定性,缺乏对预测结果可靠性的评估.为解决上述问题,本文提出了一种基于贝叶斯网络和图注意力Transformer的航空发动机剩余使用寿命概率预测方法.将图注意力机制融入Transformer的时间多头注意力模块,结合图注意力网络在空间特征提取上的优势和Transformer模型在时间特征提取的优势,实现数据特征时空关系的联合提取.同时,利用改进的贝叶斯网络度量预测不确定性,在得到剩余使用寿命预测点值的同时给出相应的置信区间.最后,通过在公开航空发动机数据集上的实验,证明了所提模型的有效性和先进性.

基于改进Transformer模型的多元时间序列预测

在无线数据传输中,环境干扰和网络拥塞导致的数据丢包和缺失问题显著影响了时间序列预测的稳定性。为了解决这个问题,提出了一种名为TFKNet的时间序列预测模型。该模型基于Transformer的多维时间序列数据预测方法,在传统Transformer模型的基础上,结合时间卷积网络(TCN)和傅立叶频率特征提取技术增强了模型对局部特征的捕捉能力和多频率特征的提取能力,引入Kernel Attention Networks(KAN)提高了模型的预测性能。实验结果表明,与Transformer、Informer、Reformer、Autoformer传统方法相比,TFKNet模型在时间序列长预测任务中预测误差MAE分别平均降低0.0522、0.1117、0.1209、0.1922。

视觉Transformer(ViT)发展综述

视觉Transformer(Vision Transformer,ViT)是基于编码器-解码器结构的Transformer改进模型,已经被成功应用于计算机视觉领域。近几年基于ViT的研究层出不穷且效果显著,基于该模型的工作已经成为计算机视觉任务的重要研究方向,因此针对近年来ViT的发展进行概述。首先,简要回顾了ViT的基本原理及迁移过程,并分析了ViT模型的结构特点和优势;然后,根据各ViT变体模型的改进特点,归纳和梳理了基于ViT的主要骨干网络变体改进方向及其代表性改进模型,包括局部性改进、结构改进、自监督、轻量化及效率改进等改进方向,并对其进行分析比较;最后,讨论了当前ViT及其改进模型仍存在的不足,对ViT未来的研究方向进行了展望。可以作为研究人员进行基于ViT骨干网络的研究时选择深度学习相关方法的一个权衡和参考。

Unraveling the significance of cobalt on transformation kinetics,crystallography and impact toughness in high-strength steels

This work reveals the significant effects of cobalt(Co)on the microstructure and impact toughness of as-quenched highstrength steels by experimental characterizations and thermo-kinetic analyses.The results show that the Co-bearing steel exhibits finer blocks and a lower ductile-brittle transition temperature than the steel without Co.Moreover,the Co-bearing steel reveals higher transformation rates at the intermediate stage with bainite volume fraction ranging from around 0.1 to 0.6.The improved impact toughness of the Co-bearing steel results from the higher dense block boundaries dominated by the V1/V2 variant pair.Furthermore,the addition of Co induces a larger transformation driving force and a lower bainite start temperature(BS),thereby contributing to the refinement of blocks and the increase of the V1/V2 variant pair.These findings would be instructive for the composition,microstructure design,and property optimization of high-strength steels.

A Cross Attention Transformer-Mixed Feedback Video Recommendation Algorithm Based on DIEN

The rapid development of short video platforms poses new challenges for traditional recommendation systems.Recommender systems typically depend on two types of user behavior feedback to construct user interest profiles:explicit feedback(interactive behavior),which significantly influences users’short-term interests,and implicit feedback(viewing time),which substantially affects their long-term interests.However,the previous model fails to distinguish between these two feedback methods,leading it to predict only the overall preferences of users based on extensive historical behavior sequences.Consequently,it cannot differentiate between users’long-term and shortterm interests,resulting in low accuracy in describing users’interest states and predicting the evolution of their interests.This paper introduces a video recommendationmodel calledCAT-MFRec(CrossAttention Transformer-Mixed Feedback Recommendation)designed to differentiate between explicit and implicit user feedback within the DIEN(Deep Interest Evolution Network)framework.This study emphasizes the separate learning of the two types of behavioral feedback,effectively integrating them through the cross-attention mechanism.Additionally,it leverages the long sequence dependence capabilities of Transformer technology to accurately construct user interest profiles and predict the evolution of user interests.Experimental results indicate that CAT-MF Rec significantly outperforms existing recommendation methods across various performance indicators.This advancement offers new theoretical and practical insights for the development of video recommendations,particularly in addressing complex and dynamic user behavior patterns.

融合快速边缘注意力的Transformer跟踪算法

针对长期目标跟踪中出现模型退化和跟踪漂移的问题,提出了一种融合快速边缘注意力的Transformer跟踪算法TransFEA(fast edge attention on Transformer)。使用ResNet-50作为Siamese网络的骨干网络,并在其每个残差块后端引入注意力网络进行特征提取,增强目标的关键信息和全局信息;边缘注意力网络(edge attention network,EA)提取模板与搜索区域的特征向量,快速注意力网络(fast attention network,FA)计算注意响应值,确定两个区域的相似度,以此调整目标位置。设计多层感知器预测边界框,避免过多超参数,使跟踪器实现了准确性与轻量化的平衡。实验结果表明,TransFEA在LaSOT数据集上成功率和准确率分别为65.3%、69.1%,运行可以达到90 FPS,提高了长期跟踪的成功率和准确率。

A Hybrid Approach for Pavement Crack Detection Using Mask R-CNN and Vision Transformer Model

Detecting pavement cracks is critical for road safety and infrastructure management.Traditional methods,relying on manual inspection and basic image processing,are time-consuming and prone to errors.Recent deep-learning(DL)methods automate crack detection,but many still struggle with variable crack patterns and environmental conditions.This study aims to address these limitations by introducing the Masker Transformer,a novel hybrid deep learning model that integrates the precise localization capabilities of Mask Region-based Convolutional Neural Network(Mask R-CNN)with the global contextual awareness of Vision Transformer(ViT).The research focuses on leveraging the strengths of both architectures to enhance segmentation accuracy and adaptability across different pavement conditions.We evaluated the performance of theMaskerTransformer against other state-of-theartmodels such asU-Net,TransformerU-Net(TransUNet),U-NetTransformer(UNETr),SwinU-NetTransformer(Swin-UNETr),You Only Look Once version 8(YoloV8),and Mask R-CNN using two benchmark datasets:Crack500 and DeepCrack.The findings reveal that the MaskerTransformer significantly outperforms the existing models,achieving the highest Dice SimilarityCoefficient(DSC),precision,recall,and F1-Score across both datasets.Specifically,the model attained a DSC of 80.04%on Crack500 and 91.37%on DeepCrack,demonstrating superior segmentation accuracy and reliability.The high precision and recall rates further substantiate its effectiveness in real-world applications,suggesting that the Masker Transformer can serve as a robust tool for automated pavement crack detection,potentially replacing more traditional methods.

Soliton Solutions of a Coupled KdV System via Backlund Transformation

In this paper we use Böcklund transformation to construct soliton solutions for a coupled KdV system.This system was first proposed by Wang in 2010.First we generalize the well-known Bäcklund transformation for the KdV equation to such coupled KdV system.Then from a trivial seed solution,we construct soliton solutions.We also give a nonlinear superposition formula,which allows us to generate multi-soliton solutions.

局部特征增强的磁Transformer重构

研究了一种基于多头自注意力与卷积特征融合的磁共振图像Transformer重构方法.采用U型网络结构,通过学习图像的多尺度特征以提升重构性能.采用深度分离卷积与多头自注意力融合的Swin Transformer结构,改善网络的特征学习能力.在CC359-Brain数据集下基于多种采样模式进行仿真实验,结果证明了该方法在提升磁共振图像重构质量与降低系统复杂度方面的有效性.

Transforming growth factor-beta 1 enhances discharge activity of cortical neurons

Transforming growth factor-beta 1(TGF-β1)has been extensively studied for its pleiotropic effects on central nervous system diseases.The neuroprotective or neurotoxic effects of TGF-β1 in specific brain areas may depend on the pathological process and cell types involved.Voltage-gated sodium channels(VGSCs)are essential ion channels for the generation of action potentials in neurons,and are involved in various neuroexcitation-related diseases.However,the effects of TGF-β1 on the functional properties of VGSCs and firing properties in cortical neurons remain unclear.In this study,we investigated the effects of TGF-β1 on VGSC function and firing properties in primary cortical neurons from mice.We found that TGF-β1 increased VGSC current density in a dose-and time-dependent manner,which was attributable to the upregulation of Nav1.3 expression.Increased VGSC current density and Nav1.3 expression were significantly abolished by preincubation with inhibitors of mitogen-activated protein kinase kinase(PD98059),p38 mitogen-activated protein kinase(SB203580),and Jun NH2-terminal kinase 1/2 inhibitor(SP600125).Interestingly,TGF-β1 significantly increased the firing threshold of action potentials but did not change their firing rate in cortical neurons.These findings suggest that TGF-β1 can increase Nav1.3 expression through activation of the ERK1/2-JNK-MAPK pathway,which leads to a decrease in the firing threshold of action potentials in cortical neurons under pathological conditions.Thus,this contributes to the occurrence and progression of neuroexcitatory-related diseases of the central nervous system.

Stage IV malignant transformation of mature cystic teratoma palliatively treated with concurrent chemoradiotherapy:A case report

BACKGROUND Malignant transformation(MT)of mature cystic teratoma(MCT)has a poor prognosis,especially in advanced cases.Concurrent chemoradiotherapy(CCRT)has an inhibitory effect on MT.CASE SUMMARY Herein,we present a case in which CCRT had a reduction effect preoperatively.A 73-year-old woman with pyelonephritis was referred to our hospital.Computed tomography revealed right hydronephrosis and a 6-cm pelvic mass.Endoscopic ultrasound-guided fine-needle biopsy(EUS-FNB)revealed squamous cell carci-noma.The patient was diagnosed with MT of MCT.Due to her poor general con-dition and renal malfunction,we selected CCRT,expecting fewer adverse effects.After CCRT,her performance status improved,and the tumor size was reduced;surgery was performed.Five months postoperatively,the patient developed dis-semination and lymph node metastases.Palliative chemotherapy was ineffective.She died 18 months after treatment initiation.CONCLUSION EUS-FNB was useful in the diagnosis of MT of MCT;CCRT suppressed the disea-se and improved quality of life.

Steel Surface Defect Detection Using Learnable Memory Vision Transformer

This study investigates the application of Learnable Memory Vision Transformers(LMViT)for detecting metal surface flaws,comparing their performance with traditional CNNs,specifically ResNet18 and ResNet50,as well as other transformer-based models including Token to Token ViT,ViT withoutmemory,and Parallel ViT.Leveraging awidely-used steel surface defect dataset,the research applies data augmentation and t-distributed stochastic neighbor embedding(t-SNE)to enhance feature extraction and understanding.These techniques mitigated overfitting,stabilized training,and improved generalization capabilities.The LMViT model achieved a test accuracy of 97.22%,significantly outperforming ResNet18(88.89%)and ResNet50(88.90%),aswell as the Token to TokenViT(88.46%),ViT without memory(87.18),and Parallel ViT(91.03%).Furthermore,LMViT exhibited superior training and validation performance,attaining a validation accuracy of 98.2%compared to 91.0%for ResNet 18,96.0%for ResNet50,and 89.12%,87.51%,and 91.21%for Token to Token ViT,ViT without memory,and Parallel ViT,respectively.The findings highlight the LMViT’s ability to capture long-range dependencies in images,an areawhere CNNs struggle due to their reliance on local receptive fields and hierarchical feature extraction.The additional transformer-based models also demonstrate improved performance in capturing complex features over CNNs,with LMViT excelling particularly at detecting subtle and complex defects,which is critical for maintaining product quality and operational efficiency in industrial applications.For instance,the LMViT model successfully identified fine scratches and minor surface irregularities that CNNs often misclassify.This study not only demonstrates LMViT’s potential for real-world defect detection but also underscores the promise of other transformer-based architectures like Token to Token ViT,ViT without memory,and Parallel ViT in industrial scenarios where complex spatial relationships are key.Future research may focus on enhancing LMViT’s computational efficiency for deployment in real-time quality control systems.

Xiamen after 20 Years of Ecological Transformation

The efforts Xiamen has made in ecological protection over the last 20 years have helped it become a beautiful and clean city.TWENTY years ago,I had the opportunity to travel to the city of Xiamen in southeastern China’s Fujian Province.Unfortunately,due to the fact that I was in a rush,I did not have the chance to see the whole city.My memory of Xiamen was its pleasant climate and unforgettable beauty-I knew if I ever got the opportunity,I would revisit this place.Twenty years went by until I was able to return to this city again.After emerging from the airport,I realized that I had grown older,while Xiamen was becoming increasingly youthful and vibrant.What exactly had given this city such vitality?

Results Involving Partial Differential Equations and Their Solution by Certain Integral Transform

In this study,we aimto investigate certain triple integral transformand its application to a class of partial differentialequations.We discuss various properties of the new transformincluding inversion, linearity, existence, scaling andshifting, etc. Then,we derive several results enfolding partial derivatives and establish amulti-convolution theorem.Further, we apply the aforementioned transform to some classical functions and many types of partial differentialequations involving heat equations,wave equations, Laplace equations, and Poisson equations aswell.Moreover,wedraw some figures to illustrate 3-D contour plots for exact solutions of some selected examples involving differentvalues in their variables.

A Comprehensive Survey of Recent Transformers in Image,Video and Diffusion Models

Transformer models have emerged as dominant networks for various tasks in computer vision compared to Convolutional Neural Networks(CNNs).The transformers demonstrate the ability to model long-range dependencies by utilizing a self-attention mechanism.This study aims to provide a comprehensive survey of recent transformerbased approaches in image and video applications,as well as diffusion models.We begin by discussing existing surveys of vision transformers and comparing them to this work.Then,we review the main components of a vanilla transformer network,including the self-attention mechanism,feed-forward network,position encoding,etc.In the main part of this survey,we review recent transformer-based models in three categories:Transformer for downstream tasks,Vision Transformer for Generation,and Vision Transformer for Segmentation.We also provide a comprehensive overview of recent transformer models for video tasks and diffusion models.We compare the performance of various hierarchical transformer networks for multiple tasks on popular benchmark datasets.Finally,we explore some future research directions to further improve the field.

一种基于Transformer编码器与LSTM的飞机轨迹预测方法

为了解决飞机目标机动数据集缺失的问题,文章利用运动学建模生成了丰富的轨迹数据集,为网络训练提供了必要的数据支持。针对现阶段轨迹预测运动学模型建立困难及时序预测方法难以提取时空特征的问题,提出了一种结合Transformer编码器和长短期记忆网络(Long Short Term Memory,LSTM)的飞机目标轨迹预测方法,即Transformer-Encoder-LSTM模型。新模型可同时提供LSTM和Transformer编码器模块的补充历史信息和基于注意力的信息表示,提高了模型能力。通过与一些经典神经网络模型进行对比分析,发现在数据集上,新方法的平均位移误差减小到0.22,显著优于CNN-LSTMAttention模型的0.35。相比其他网络,该算法能够提取复杂轨迹中的隐藏特征,在面对飞机连续转弯、大机动转弯的复杂轨迹时,能够保证模型的鲁棒性,提升了对于复杂轨迹预测的准确性。