基于Transformer模型的时序数据预测方法综述

时序数据预测(TSF)是指通过分析历史数据的趋势性、季节性等潜在信息,预测未来时间点或时间段的数值和趋势。时序数据由传感器生成,在金融、医疗、能源、交通、气象等众多领域都发挥着重要作用。随着物联网传感器的发展,海量的时序数据难以使用传统的机器学习解决,而Transformer在自然语言处理和计算机视觉等领域的诸多任务表现优秀,学者们利用Transformer模型有效捕获长期依赖关系,使得时序数据预测任务取得了飞速发展。综述了基于Transformer模型的时序数据预测方法,按时间梳理了时序数据预测的发展进程,系统介绍了时序数据预处理过程和方法,介绍了常用的时序预测评价指标和数据集。以算法框架为研究内容系统阐述了基于Transformer的各类模型在TSF任务中的应用方法和工作原理。通过实验对比了各个模型的性能、优点和局限性,并对实验结果展开了分析与讨论。结合Transformer模型在时序数据预测任务中现有工作存在的挑战提出了该方向未来发展趋势。

局部特征增强的磁Transformer重构

研究了一种基于多头自注意力与卷积特征融合的磁共振图像Transformer重构方法.采用U型网络结构,通过学习图像的多尺度特征以提升重构性能.采用深度分离卷积与多头自注意力融合的Swin Transformer结构,改善网络的特征学习能力.在CC359-Brain数据集下基于多种采样模式进行仿真实验,结果证明了该方法在提升磁共振图像重构质量与降低系统复杂度方面的有效性.

图像处理中CNN与视觉Transformer混合模型研究综述

卷积神经网络(CNN)与视觉Transformer是目前图像处理领域中两大重要的深度学习模型,两者经过多年来不断的研究与进步,已在该领域取得了非凡的成就。近些年来,CNN与视觉Transformer的混合模型正在逐步兴起,广泛的研究不断克服两种模型存在的弱项,高效地发挥出各自的亮点,在图像处理任务中表现出优异的效果。基于CNN与视觉Transformer混合模型进行深入阐述。总体概述了CNN与Vision Transformer模型的架构和优缺点,并总结混合模型的概念及优势。围绕串行结构融合方式、并行结构融合方式、层级交叉结构融合方式以及其他融合方式等四个方面全面回顾梳理了混合模型的研究现状和实际进展,并针对各种融合方式的主要代表模型进行总结与剖析,从多方面对典型混合模型进行评价对比。多角度叙述了混合模型在图像识别、图像分类、目标检测和图像分割等实际图像处理特定领域中应用研究,展现出混合模型在具体实践中的适用性和高效性。深入分析混合模型未来研究方向,并为后续该模型在图像处理中的研究与应用提出展望。

基于改进Transformer模型的多元时间序列预测

在无线数据传输中,环境干扰和网络拥塞导致的数据丢包和缺失问题显著影响了时间序列预测的稳定性。为了解决这个问题,提出了一种名为TFKNet的时间序列预测模型。该模型基于Transformer的多维时间序列数据预测方法,在传统Transformer模型的基础上,结合时间卷积网络(TCN)和傅立叶频率特征提取技术增强了模型对局部特征的捕捉能力和多频率特征的提取能力,引入Kernel Attention Networks(KAN)提高了模型的预测性能。实验结果表明,与Transformer、Informer、Reformer、Autoformer传统方法相比,TFKNet模型在时间序列长预测任务中预测误差MAE分别平均降低0.0522、0.1117、0.1209、0.1922。

ViTAU:基于Vision transformer和面部动作单元的面瘫识别与分析

面部神经麻痹(Facial nerve paralysis,FNP),通常称为贝尔氏麻痹或面瘫,对患者的日常生活和心理健康产生显著影响,面瘫的及时识别和诊断对于患者的早期治疗和康复至关重要.随着深度学习和计算机视觉技术的快速发展,面瘫的自动识别变得可行,为诊断提供了一种更准确和客观的方式.目前的研究主要集中关注面部的整体变化,而忽略了面部细节的重要性.面部不同部位对识别结果的影响力并不相同,这些研究尚未对面部各个区域进行细致区分和分析.本项研究引入结合Vision transformer(ViT)模型和动作单元(Action unit,AU)区域检测网络的创新性方法用于面瘫的自动识别及区域分析.ViT模型通过自注意力机制精准识别是否面瘫,同时,基于AU的策略从StyleGAN2模型提取的特征图中,利用金字塔卷积神经网络分析受影响区域.这一综合方法在YouTube Facial Palsy(YFP)和经过扩展的Cohn Kanade(CK+)数据集上的实验中分别达到99.4%的面瘫识别准确率和81.36%的面瘫区域识别准确率.通过与最新方法的对比,实验结果展示了所提的自动面瘫识别方法的有效性.

基于贝叶斯图注意力Transformer的航空发动机剩余使用寿命概率预测

航空发动机作为飞机的心脏,其健康状态对飞机的安全飞行至关重要.深度学习强大的数据挖掘能力,为通过海量历史数据预测航空发动机的剩余使用寿命提供了新方法.然而,传统基于深度学习的方法大都关注于挖掘数据在时间上的关联,而忽略了多个传感器监测数据之间复杂的非欧氏空间关系.此外,少有研究考虑数据或者预测过程本身具有的不确定性,缺乏对预测结果可靠性的评估.为解决上述问题,本文提出了一种基于贝叶斯网络和图注意力Transformer的航空发动机剩余使用寿命概率预测方法.将图注意力机制融入Transformer的时间多头注意力模块,结合图注意力网络在空间特征提取上的优势和Transformer模型在时间特征提取的优势,实现数据特征时空关系的联合提取.同时,利用改进的贝叶斯网络度量预测不确定性,在得到剩余使用寿命预测点值的同时给出相应的置信区间.最后,通过在公开航空发动机数据集上的实验,证明了所提模型的有效性和先进性.

融合快速边缘注意力的Transformer跟踪算法

针对长期目标跟踪中出现模型退化和跟踪漂移的问题,提出了一种融合快速边缘注意力的Transformer跟踪算法TransFEA(fast edge attention on Transformer)。使用ResNet-50作为Siamese网络的骨干网络,并在其每个残差块后端引入注意力网络进行特征提取,增强目标的关键信息和全局信息;边缘注意力网络(edge attention network,EA)提取模板与搜索区域的特征向量,快速注意力网络(fast attention network,FA)计算注意响应值,确定两个区域的相似度,以此调整目标位置。设计多层感知器预测边界框,避免过多超参数,使跟踪器实现了准确性与轻量化的平衡。实验结果表明,TransFEA在LaSOT数据集上成功率和准确率分别为65.3%、69.1%,运行可以达到90 FPS,提高了长期跟踪的成功率和准确率。

视觉Transformer(ViT)发展综述

视觉Transformer(Vision Transformer,ViT)是基于编码器-解码器结构的Transformer改进模型,已经被成功应用于计算机视觉领域。近几年基于ViT的研究层出不穷且效果显著,基于该模型的工作已经成为计算机视觉任务的重要研究方向,因此针对近年来ViT的发展进行概述。首先,简要回顾了ViT的基本原理及迁移过程,并分析了ViT模型的结构特点和优势;然后,根据各ViT变体模型的改进特点,归纳和梳理了基于ViT的主要骨干网络变体改进方向及其代表性改进模型,包括局部性改进、结构改进、自监督、轻量化及效率改进等改进方向,并对其进行分析比较;最后,讨论了当前ViT及其改进模型仍存在的不足,对ViT未来的研究方向进行了展望。可以作为研究人员进行基于ViT骨干网络的研究时选择深度学习相关方法的一个权衡和参考。

A Cross Attention Transformer-Mixed Feedback Video Recommendation Algorithm Based on DIEN

The rapid development of short video platforms poses new challenges for traditional recommendation systems.Recommender systems typically depend on two types of user behavior feedback to construct user interest profiles:explicit feedback(interactive behavior),which significantly influences users’short-term interests,and implicit feedback(viewing time),which substantially affects their long-term interests.However,the previous model fails to distinguish between these two feedback methods,leading it to predict only the overall preferences of users based on extensive historical behavior sequences.Consequently,it cannot differentiate between users’long-term and shortterm interests,resulting in low accuracy in describing users’interest states and predicting the evolution of their interests.This paper introduces a video recommendationmodel calledCAT-MFRec(CrossAttention Transformer-Mixed Feedback Recommendation)designed to differentiate between explicit and implicit user feedback within the DIEN(Deep Interest Evolution Network)framework.This study emphasizes the separate learning of the two types of behavioral feedback,effectively integrating them through the cross-attention mechanism.Additionally,it leverages the long sequence dependence capabilities of Transformer technology to accurately construct user interest profiles and predict the evolution of user interests.Experimental results indicate that CAT-MF Rec significantly outperforms existing recommendation methods across various performance indicators.This advancement offers new theoretical and practical insights for the development of video recommendations,particularly in addressing complex and dynamic user behavior patterns.

Unraveling the significance of cobalt on transformation kinetics,crystallography and impact toughness in high-strength steels

This work reveals the significant effects of cobalt(Co)on the microstructure and impact toughness of as-quenched highstrength steels by experimental characterizations and thermo-kinetic analyses.The results show that the Co-bearing steel exhibits finer blocks and a lower ductile-brittle transition temperature than the steel without Co.Moreover,the Co-bearing steel reveals higher transformation rates at the intermediate stage with bainite volume fraction ranging from around 0.1 to 0.6.The improved impact toughness of the Co-bearing steel results from the higher dense block boundaries dominated by the V1/V2 variant pair.Furthermore,the addition of Co induces a larger transformation driving force and a lower bainite start temperature(BS),thereby contributing to the refinement of blocks and the increase of the V1/V2 variant pair.These findings would be instructive for the composition,microstructure design,and property optimization of high-strength steels.

Soliton Solutions of a Coupled KdV System via Backlund Transformation

In this paper we use Böcklund transformation to construct soliton solutions for a coupled KdV system.This system was first proposed by Wang in 2010.First we generalize the well-known Bäcklund transformation for the KdV equation to such coupled KdV system.Then from a trivial seed solution,we construct soliton solutions.We also give a nonlinear superposition formula,which allows us to generate multi-soliton solutions.

A Hybrid Approach for Pavement Crack Detection Using Mask R-CNN and Vision Transformer Model

Detecting pavement cracks is critical for road safety and infrastructure management.Traditional methods,relying on manual inspection and basic image processing,are time-consuming and prone to errors.Recent deep-learning(DL)methods automate crack detection,but many still struggle with variable crack patterns and environmental conditions.This study aims to address these limitations by introducing the Masker Transformer,a novel hybrid deep learning model that integrates the precise localization capabilities of Mask Region-based Convolutional Neural Network(Mask R-CNN)with the global contextual awareness of Vision Transformer(ViT).The research focuses on leveraging the strengths of both architectures to enhance segmentation accuracy and adaptability across different pavement conditions.We evaluated the performance of theMaskerTransformer against other state-of-theartmodels such asU-Net,TransformerU-Net(TransUNet),U-NetTransformer(UNETr),SwinU-NetTransformer(Swin-UNETr),You Only Look Once version 8(YoloV8),and Mask R-CNN using two benchmark datasets:Crack500 and DeepCrack.The findings reveal that the MaskerTransformer significantly outperforms the existing models,achieving the highest Dice SimilarityCoefficient(DSC),precision,recall,and F1-Score across both datasets.Specifically,the model attained a DSC of 80.04%on Crack500 and 91.37%on DeepCrack,demonstrating superior segmentation accuracy and reliability.The high precision and recall rates further substantiate its effectiveness in real-world applications,suggesting that the Masker Transformer can serve as a robust tool for automated pavement crack detection,potentially replacing more traditional methods.

Stage IV malignant transformation of mature cystic teratoma palliatively treated with concurrent chemoradiotherapy:A case report

BACKGROUND Malignant transformation(MT)of mature cystic teratoma(MCT)has a poor prognosis,especially in advanced cases.Concurrent chemoradiotherapy(CCRT)has an inhibitory effect on MT.CASE SUMMARY Herein,we present a case in which CCRT had a reduction effect preoperatively.A 73-year-old woman with pyelonephritis was referred to our hospital.Computed tomography revealed right hydronephrosis and a 6-cm pelvic mass.Endoscopic ultrasound-guided fine-needle biopsy(EUS-FNB)revealed squamous cell carci-noma.The patient was diagnosed with MT of MCT.Due to her poor general con-dition and renal malfunction,we selected CCRT,expecting fewer adverse effects.After CCRT,her performance status improved,and the tumor size was reduced;surgery was performed.Five months postoperatively,the patient developed dis-semination and lymph node metastases.Palliative chemotherapy was ineffective.She died 18 months after treatment initiation.CONCLUSION EUS-FNB was useful in the diagnosis of MT of MCT;CCRT suppressed the disea-se and improved quality of life.

Transforming growth factor-beta 1 enhances discharge activity of cortical neurons

Transforming growth factor-beta 1(TGF-β1)has been extensively studied for its pleiotropic effects on central nervous system diseases.The neuroprotective or neurotoxic effects of TGF-β1 in specific brain areas may depend on the pathological process and cell types involved.Voltage-gated sodium channels(VGSCs)are essential ion channels for the generation of action potentials in neurons,and are involved in various neuroexcitation-related diseases.However,the effects of TGF-β1 on the functional properties of VGSCs and firing properties in cortical neurons remain unclear.In this study,we investigated the effects of TGF-β1 on VGSC function and firing properties in primary cortical neurons from mice.We found that TGF-β1 increased VGSC current density in a dose-and time-dependent manner,which was attributable to the upregulation of Nav1.3 expression.Increased VGSC current density and Nav1.3 expression were significantly abolished by preincubation with inhibitors of mitogen-activated protein kinase kinase(PD98059),p38 mitogen-activated protein kinase(SB203580),and Jun NH2-terminal kinase 1/2 inhibitor(SP600125).Interestingly,TGF-β1 significantly increased the firing threshold of action potentials but did not change their firing rate in cortical neurons.These findings suggest that TGF-β1 can increase Nav1.3 expression through activation of the ERK1/2-JNK-MAPK pathway,which leads to a decrease in the firing threshold of action potentials in cortical neurons under pathological conditions.Thus,this contributes to the occurrence and progression of neuroexcitatory-related diseases of the central nervous system.

Steel Surface Defect Detection Using Learnable Memory Vision Transformer

This study investigates the application of Learnable Memory Vision Transformers(LMViT)for detecting metal surface flaws,comparing their performance with traditional CNNs,specifically ResNet18 and ResNet50,as well as other transformer-based models including Token to Token ViT,ViT withoutmemory,and Parallel ViT.Leveraging awidely-used steel surface defect dataset,the research applies data augmentation and t-distributed stochastic neighbor embedding(t-SNE)to enhance feature extraction and understanding.These techniques mitigated overfitting,stabilized training,and improved generalization capabilities.The LMViT model achieved a test accuracy of 97.22%,significantly outperforming ResNet18(88.89%)and ResNet50(88.90%),aswell as the Token to TokenViT(88.46%),ViT without memory(87.18),and Parallel ViT(91.03%).Furthermore,LMViT exhibited superior training and validation performance,attaining a validation accuracy of 98.2%compared to 91.0%for ResNet 18,96.0%for ResNet50,and 89.12%,87.51%,and 91.21%for Token to Token ViT,ViT without memory,and Parallel ViT,respectively.The findings highlight the LMViT’s ability to capture long-range dependencies in images,an areawhere CNNs struggle due to their reliance on local receptive fields and hierarchical feature extraction.The additional transformer-based models also demonstrate improved performance in capturing complex features over CNNs,with LMViT excelling particularly at detecting subtle and complex defects,which is critical for maintaining product quality and operational efficiency in industrial applications.For instance,the LMViT model successfully identified fine scratches and minor surface irregularities that CNNs often misclassify.This study not only demonstrates LMViT’s potential for real-world defect detection but also underscores the promise of other transformer-based architectures like Token to Token ViT,ViT without memory,and Parallel ViT in industrial scenarios where complex spatial relationships are key.Future research may focus on enhancing LMViT’s computational efficiency for deployment in real-time quality control systems.

基于Depth-wise卷积和视觉Transformer的图像分类模型

图像分类作为一种常见的视觉识别任务,有着广阔的应用场景。在处理图像分类问题时,传统的方法通常使用卷积神经网络,然而,卷积网络的感受野有限,难以建模图像的全局关系表示,导致分类精度低,难以处理复杂多样的图像数据。为了对全局关系进行建模,一些研究者将Transformer应用于图像分类任务,但为了满足Transformer的序列化和并行化要求,需要将图像分割成大小相等、互不重叠的图像块,破坏了相邻图像数据块之间的局部信息。此外,由于Transformer具有较少的先验知识,模型往往需要在大规模数据集上进行预训练,因此计算复杂度较高。为了同时建模图像相邻块之间的局部信息并充分利用图像的全局信息,提出了一种基于Depth-wise卷积的视觉Transformer(Efficient Pyramid Vision Transformer,EPVT)模型。EPVT模型可以实现以较低的计算成本提取相邻图像块之间的局部和全局信息。EPVT模型主要包含3个关键组件:局部感知模块(Local Perceptron Module,LPM)、空间信息融合模块(Spatial Information Fusion,SIF)和“+卷积前馈神经网络(Convolution Feed-forward Network,CFFN)。LPM模块用于捕获图像的局部相关性;SIF模块用于融合相邻图像块之间的局部信息,并利用不同图像块之间的远距离依赖关系,提升模型的特征表达能力,使模型学习到输出特征在不同维度下的语义信息;CFFN模块用于编码位置信息和重塑张量。在图像分类数据集ImageNet-1K上,所提模型优于现有的同等规模的视觉Transformer分类模型,取得了82.6%的分类准确度,证明了该模型在大规模数据集上具有竞争力。

CNN-Transformer特征融合多目标跟踪算法

在卷积神经网络(CNN)中,卷积运算能高效地提取目标的局部特征,却难以捕获全局表示;而在视觉Transformer中,注意力机制可以捕获长距离的特征依赖,但会忽略局部特征细节。针对以上问题,提出一种基于CNN-Transformer双分支主干网络进行特征提取和融合的多目标跟踪算法CTMOT(CNN-transformer multi-object tracking)。使用基于CNN和Transformer双分支并行的主干网络分别提取图像的局部和全局特征。使用双向桥接模块(two-way braidge module,TBM)对两种特征进行充分融合。将融合后的特征输入两组并行的解码器进行处理。将解码器输出的检测框和跟踪框进行匹配,完成多目标跟踪任务。在多目标跟踪数据集MOT17、MOT20、KITTI以及UADETRAC上进行评估,CTMOT算法的MOTP和IDs指标在四个数据集上均达到了SOTA效果,MOTA指标分别达到了76.4%、66.3%、92.36%和88.57%,在MOT数据集上与SOTA方法效果相当,在KITTI数据集上达到SOTA效果。由于同时完成目标检测和关联,能够端到端进行目标跟踪,跟踪速度可达35 FPS,表明CTMOT算法在跟踪的实时性和准确性上达到了较好的平衡,具有较大潜力。

基于RoBERTa和图增强Transformer的序列推荐方法

自推荐系统出现以来,有限的数据信息就一直制约着推荐算法的进一步发展。为降低数据稀疏性的影响,增强非评分数据的利用率,基于神经网络的文本推荐模型相继被提出,但主流的卷积或循环神经网络在文本语义理解和长距离关系捕捉方面存在明显劣势。为了更好地挖掘用户与商品之间的深层潜在特征,进一步提高推荐质量,提出一种基于Ro BERTa和图增强Transformer的序列推荐(RGT)模型。引入评论文本数据,首先利用预训练的Ro BERTa模型捕获评论文本中的字词语义特征,初步建模用户的个性化兴趣,然后根据用户与商品的历史交互信息,构建具有时序特性的商品关联图注意力机制网络模型,通过图增强Transformer的方法将图模型学习到的各个商品的特征表示以序列的形式输入Transformer编码层,最后将得到的输出向量与之前捕获的语义表征以及计算得到的商品关联图的全图表征输入全连接层,以捕获用户全局的兴趣偏好,实现用户对商品的预测评分。在3组真实亚马逊公开数据集上的实验结果表明,与Deep FM、Conv MF等经典文本推荐模型相比,RGT模型在均方根误差(RMSE)和平均绝对误差(MAE)2种指标上有显著提升,相较于最优对比模型最高分别提升4.7%和5.3%。

基于Transformer和自适应特征融合的矿井低照度图像亮度提升和细节增强方法

高质量矿井影像为矿山安全生产提供保障,也有利于提高后续图像分析技术的性能。矿井影像受低照度环境的影响,易出现亮度低,照度不均,颜色失真,细节信息丢失严重等问题。针对上述问题,提出一种基于Transformer和自适应特征融合的矿井低照度图像亮度提升和细节增强方法。基于生成对抗思想搭建生成对抗式主体模型框架,使用目标图像域而非单一参考图像驱动判别器监督生成器的训练,实现对低照度图像的充分增强;基于特征表示学习理论搭建特征编码器,将图像解耦为亮度分量和反射分量,避免图像增强过程中亮度与颜色特征相互影响从而导致颜色失真问题;设计CEM-Transformer Encoder通过捕获全局上下文关系和提取局部区域特征,能够充分提升整体图像亮度并消除局部区域照度不均;在反射分量增强过程中,使用结合CEM-Cross-Transformer Encoder的跳跃连接将低级特征与深层网络处特征进行自适应融合,能够有效避免细节特征丢失,并在编码网络中添加ECA-Net,提高浅层网络的特征提取效率。制作矿井低照度图像数据集为矿井低照度图像增强任务提供数据资源。试验显示,在矿井低照度图像数据集和公共数据集中,与5种先进的低照度图像增强算法相比,该算法增强图像的质量指标PSNR、SSIM、VIF平均提高了16.564%,10.998%,16.226%和14.438%,10.888%,14.948%,证明该算法能够有效提升整体图像亮度,消除照度不均,避免颜色失真和细节丢失,实现矿井低照度图像增强。

基于残差U-Net和自注意力Transformer编码器的磁场预测方法

利用有限元方法对几何结构复杂的电机和变压器进行磁场分析,存在仿真时间长且无法复用的问题。因此,该文提出一种基于残差U-Net和自注意力Transformer编码器的磁场预测方法。首先建立永磁同步电机(PMSM)和非晶合金变压器(AMT)有限元模型,得到深度学习训练所需的数据集;然后将Transformer模块与U-Net模型结合,并引入短残差机制建立ResUnet-Transformer模型,通过预测图像的像素实现磁场预测;最后通过Targeted Dropout算法和动态学习率调整策略对模型进行优化,解决拟合问题并提高预测精度。计算实例证明,ResUnet-Transformer模型在PMSM和AMT数据集上测试集的平均绝对百分比误差(MAPE)均小于1%,且仅需500组样本。该文提出的磁场预测方法能减少实际工况和多工况下精细模拟和拓扑优化的时间和资源消耗,亦是虚拟传感器乃至数字孪生的关键实现方法之一。