A Cross Attention Transformer-Mixed Feedback Video Recommendation Algorithm Based on DIEN

The rapid development of short video platforms poses new challenges for traditional recommendation systems.Recommender systems typically depend on two types of user behavior feedback to construct user interest profiles:explicit feedback(interactive behavior),which significantly influences users’short-term interests,and implicit feedback(viewing time),which substantially affects their long-term interests.However,the previous model fails to distinguish between these two feedback methods,leading it to predict only the overall preferences of users based on extensive historical behavior sequences.Consequently,it cannot differentiate between users’long-term and shortterm interests,resulting in low accuracy in describing users’interest states and predicting the evolution of their interests.This paper introduces a video recommendationmodel calledCAT-MFRec(CrossAttention Transformer-Mixed Feedback Recommendation)designed to differentiate between explicit and implicit user feedback within the DIEN(Deep Interest Evolution Network)framework.This study emphasizes the separate learning of the two types of behavioral feedback,effectively integrating them through the cross-attention mechanism.Additionally,it leverages the long sequence dependence capabilities of Transformer technology to accurately construct user interest profiles and predict the evolution of user interests.Experimental results indicate that CAT-MF Rec significantly outperforms existing recommendation methods across various performance indicators.This advancement offers new theoretical and practical insights for the development of video recommendations,particularly in addressing complex and dynamic user behavior patterns.

Unraveling the significance of cobalt on transformation kinetics,crystallography and impact toughness in high-strength steels

This work reveals the significant effects of cobalt(Co)on the microstructure and impact toughness of as-quenched highstrength steels by experimental characterizations and thermo-kinetic analyses.The results show that the Co-bearing steel exhibits finer blocks and a lower ductile-brittle transition temperature than the steel without Co.Moreover,the Co-bearing steel reveals higher transformation rates at the intermediate stage with bainite volume fraction ranging from around 0.1 to 0.6.The improved impact toughness of the Co-bearing steel results from the higher dense block boundaries dominated by the V1/V2 variant pair.Furthermore,the addition of Co induces a larger transformation driving force and a lower bainite start temperature(BS),thereby contributing to the refinement of blocks and the increase of the V1/V2 variant pair.These findings would be instructive for the composition,microstructure design,and property optimization of high-strength steels.

Soliton Solutions of a Coupled KdV System via Backlund Transformation

In this paper we use Böcklund transformation to construct soliton solutions for a coupled KdV system.This system was first proposed by Wang in 2010.First we generalize the well-known Bäcklund transformation for the KdV equation to such coupled KdV system.Then from a trivial seed solution,we construct soliton solutions.We also give a nonlinear superposition formula,which allows us to generate multi-soliton solutions.

A Hybrid Approach for Pavement Crack Detection Using Mask R-CNN and Vision Transformer Model

Detecting pavement cracks is critical for road safety and infrastructure management.Traditional methods,relying on manual inspection and basic image processing,are time-consuming and prone to errors.Recent deep-learning(DL)methods automate crack detection,but many still struggle with variable crack patterns and environmental conditions.This study aims to address these limitations by introducing the Masker Transformer,a novel hybrid deep learning model that integrates the precise localization capabilities of Mask Region-based Convolutional Neural Network(Mask R-CNN)with the global contextual awareness of Vision Transformer(ViT).The research focuses on leveraging the strengths of both architectures to enhance segmentation accuracy and adaptability across different pavement conditions.We evaluated the performance of theMaskerTransformer against other state-of-theartmodels such asU-Net,TransformerU-Net(TransUNet),U-NetTransformer(UNETr),SwinU-NetTransformer(Swin-UNETr),You Only Look Once version 8(YoloV8),and Mask R-CNN using two benchmark datasets:Crack500 and DeepCrack.The findings reveal that the MaskerTransformer significantly outperforms the existing models,achieving the highest Dice SimilarityCoefficient(DSC),precision,recall,and F1-Score across both datasets.Specifically,the model attained a DSC of 80.04%on Crack500 and 91.37%on DeepCrack,demonstrating superior segmentation accuracy and reliability.The high precision and recall rates further substantiate its effectiveness in real-world applications,suggesting that the Masker Transformer can serve as a robust tool for automated pavement crack detection,potentially replacing more traditional methods.

Stage IV malignant transformation of mature cystic teratoma palliatively treated with concurrent chemoradiotherapy:A case report

BACKGROUND Malignant transformation(MT)of mature cystic teratoma(MCT)has a poor prognosis,especially in advanced cases.Concurrent chemoradiotherapy(CCRT)has an inhibitory effect on MT.CASE SUMMARY Herein,we present a case in which CCRT had a reduction effect preoperatively.A 73-year-old woman with pyelonephritis was referred to our hospital.Computed tomography revealed right hydronephrosis and a 6-cm pelvic mass.Endoscopic ultrasound-guided fine-needle biopsy(EUS-FNB)revealed squamous cell carci-noma.The patient was diagnosed with MT of MCT.Due to her poor general con-dition and renal malfunction,we selected CCRT,expecting fewer adverse effects.After CCRT,her performance status improved,and the tumor size was reduced;surgery was performed.Five months postoperatively,the patient developed dis-semination and lymph node metastases.Palliative chemotherapy was ineffective.She died 18 months after treatment initiation.CONCLUSION EUS-FNB was useful in the diagnosis of MT of MCT;CCRT suppressed the disea-se and improved quality of life.

Transforming growth factor-beta 1 enhances discharge activity of cortical neurons

Transforming growth factor-beta 1(TGF-β1)has been extensively studied for its pleiotropic effects on central nervous system diseases.The neuroprotective or neurotoxic effects of TGF-β1 in specific brain areas may depend on the pathological process and cell types involved.Voltage-gated sodium channels(VGSCs)are essential ion channels for the generation of action potentials in neurons,and are involved in various neuroexcitation-related diseases.However,the effects of TGF-β1 on the functional properties of VGSCs and firing properties in cortical neurons remain unclear.In this study,we investigated the effects of TGF-β1 on VGSC function and firing properties in primary cortical neurons from mice.We found that TGF-β1 increased VGSC current density in a dose-and time-dependent manner,which was attributable to the upregulation of Nav1.3 expression.Increased VGSC current density and Nav1.3 expression were significantly abolished by preincubation with inhibitors of mitogen-activated protein kinase kinase(PD98059),p38 mitogen-activated protein kinase(SB203580),and Jun NH2-terminal kinase 1/2 inhibitor(SP600125).Interestingly,TGF-β1 significantly increased the firing threshold of action potentials but did not change their firing rate in cortical neurons.These findings suggest that TGF-β1 can increase Nav1.3 expression through activation of the ERK1/2-JNK-MAPK pathway,which leads to a decrease in the firing threshold of action potentials in cortical neurons under pathological conditions.Thus,this contributes to the occurrence and progression of neuroexcitatory-related diseases of the central nervous system.

Steel Surface Defect Detection Using Learnable Memory Vision Transformer

This study investigates the application of Learnable Memory Vision Transformers(LMViT)for detecting metal surface flaws,comparing their performance with traditional CNNs,specifically ResNet18 and ResNet50,as well as other transformer-based models including Token to Token ViT,ViT withoutmemory,and Parallel ViT.Leveraging awidely-used steel surface defect dataset,the research applies data augmentation and t-distributed stochastic neighbor embedding(t-SNE)to enhance feature extraction and understanding.These techniques mitigated overfitting,stabilized training,and improved generalization capabilities.The LMViT model achieved a test accuracy of 97.22%,significantly outperforming ResNet18(88.89%)and ResNet50(88.90%),aswell as the Token to TokenViT(88.46%),ViT without memory(87.18),and Parallel ViT(91.03%).Furthermore,LMViT exhibited superior training and validation performance,attaining a validation accuracy of 98.2%compared to 91.0%for ResNet 18,96.0%for ResNet50,and 89.12%,87.51%,and 91.21%for Token to Token ViT,ViT without memory,and Parallel ViT,respectively.The findings highlight the LMViT’s ability to capture long-range dependencies in images,an areawhere CNNs struggle due to their reliance on local receptive fields and hierarchical feature extraction.The additional transformer-based models also demonstrate improved performance in capturing complex features over CNNs,with LMViT excelling particularly at detecting subtle and complex defects,which is critical for maintaining product quality and operational efficiency in industrial applications.For instance,the LMViT model successfully identified fine scratches and minor surface irregularities that CNNs often misclassify.This study not only demonstrates LMViT’s potential for real-world defect detection but also underscores the promise of other transformer-based architectures like Token to Token ViT,ViT without memory,and Parallel ViT in industrial scenarios where complex spatial relationships are key.Future research may focus on enhancing LMViT’s computational efficiency for deployment in real-time quality control systems.

基于Depth-wise卷积和视觉Transformer的图像分类模型

图像分类作为一种常见的视觉识别任务,有着广阔的应用场景。在处理图像分类问题时,传统的方法通常使用卷积神经网络,然而,卷积网络的感受野有限,难以建模图像的全局关系表示,导致分类精度低,难以处理复杂多样的图像数据。为了对全局关系进行建模,一些研究者将Transformer应用于图像分类任务,但为了满足Transformer的序列化和并行化要求,需要将图像分割成大小相等、互不重叠的图像块,破坏了相邻图像数据块之间的局部信息。此外,由于Transformer具有较少的先验知识,模型往往需要在大规模数据集上进行预训练,因此计算复杂度较高。为了同时建模图像相邻块之间的局部信息并充分利用图像的全局信息,提出了一种基于Depth-wise卷积的视觉Transformer(Efficient Pyramid Vision Transformer,EPVT)模型。EPVT模型可以实现以较低的计算成本提取相邻图像块之间的局部和全局信息。EPVT模型主要包含3个关键组件:局部感知模块(Local Perceptron Module,LPM)、空间信息融合模块(Spatial Information Fusion,SIF)和“+卷积前馈神经网络(Convolution Feed-forward Network,CFFN)。LPM模块用于捕获图像的局部相关性;SIF模块用于融合相邻图像块之间的局部信息,并利用不同图像块之间的远距离依赖关系,提升模型的特征表达能力,使模型学习到输出特征在不同维度下的语义信息;CFFN模块用于编码位置信息和重塑张量。在图像分类数据集ImageNet-1K上,所提模型优于现有的同等规模的视觉Transformer分类模型,取得了82.6%的分类准确度,证明了该模型在大规模数据集上具有竞争力。

FMA-DETR:一种无编码器的Transformer目标检测方法

DETR是第一个将Transformer应用于目标检测的视觉模型。在DETR结构中,Transformer编码器对已高度编码的图像特征进行再编码,这在一定程度上导致了网络功能的重复。此外,由于Transformer编码器具有多层深度堆叠的结构和巨大的参数量,导致网络优化变得困难,模型收敛速度缓慢。本文设计了一种无编码器的Transformer目标检测网络模型。由于不需要引入Transformer编码器,本文的模型比DETR参数量更小、计算量更低、模型收敛速度更快。但是,直接去除Transformer编码器将降低网络的表达能力,导致Transformer解码器无法从数量庞大的图像特征中关注到包含目标的图像特征,从而使检测性能大幅降低。为了缓解这个问题,本文提出了一种混合特征注意力(fusion-feature mixing attention,FMA)机制,它通过自适应特征混合和通道交叉注意力弥补检测网络特征表达能力的下降,将其应用于Transformer解码器可以减轻由于去除Transformer编码器带来的性能降低。在MS-COCO数据集上,本文网络模型(称为FMA-DETR)实现了与DETR相近的性能表现,同时本文的模型拥有更快的收敛速度、更小的参数量以及更低的计算量。本文还进行了大量消融实验来验证所提出方法的有效性。

基于Transformer和动态3D卷积的多源遥感图像分类

多源遥感数据具有互补性和协同性,近年来,基于深度学习的方法已经在多源遥感图像分类中取得了一定进展,但当前方法仍面临关键难题,如多源遥感图像特征表达不一致,融合困难,基于静态推理范式的神经网络缺乏对不同类别地物的适应性。为解决上述问题,提出了基于跨模态Transformer和多尺度动态3D卷积的多源遥感图像分类模型。为提高多源特征表达的一致性,设计了基于Transformer的融合模块,借助其强大的注意力建模能力挖掘高光谱和LiDAR数据特征之间的相互作用;为提高特征提取方法对不同地物类别的适应性,设计了多尺度动态3D卷积模块,将输入特征的多尺度信息融入卷积核的调制,提高卷积操作对不同地物的适应性。采用多源遥感数据集Houston和Trento对所提方法进行验证,实验结果表明:所提方法在Houston和Trento数据集上总体准确率分别达到94.60%和98.21%,相比MGA-MFN等主流方法,总体准确率分别至少提升0.97%和0.25%,验证了所提方法可有效提升多源遥感图像分类的准确率。

基于Contextual Transformer的自动驾驶单目3D目标检测

针对当前单目3D目标检测中存在的漏检和多尺度目标检测效果不佳的问题,提出了一种基于Contextual Transformer的自动驾驶单目3D目标检测算法(CM-RTM3D)。在ResNet-50网络中引入Contextual Transformer(CoT),构建ResNet-Transformer架构以提取特征。设计多尺度空间感知模块(MSP),通过尺度空间响应操作改善浅层特征的丢失情况,嵌入沿水平和竖直两个空间方向的坐标注意力机制(CA),使用softmax函数生成各尺度的重要性软权重。在偏移损失中采用Huber损失函数代替L1损失函数。实验结果表明:在KITTI自动驾驶数据集上,相较于RTM3D算法,该算法在简单、中等、困难三个难度级别下,AP3D分别提升了4.84、3.82、5.36个百分点,APBEV分别提升了4.75、6.26、3.56个百分点。

基于Transformer改进的YOLOv5+DeepSORT的车辆跟踪算法

针对传统目标检测跟踪算法检测精度低、全局感知能力差、对遮挡和小目标物体的识别能力差等问题,提出了一种基于轻量化Transformer改进的YOLOv5和DeepSORT算法的车辆跟踪方法。首先,利用EfficientFormerV2模型改进YOLOv5算法模型,增强车辆的目标检测能力;然后,利用移位窗口(Swin)模型的优点改进DeepSORT多目标跟踪算法中的重识别(Re-Identification)模块,提高车辆的跟踪能力和精度;最后,通过数据集KITTI和VeRi开展对比试验和消融实验。结果表明,在复杂工况下,该方法的性能在车辆遮挡和小目标识别方面显著提高,平均准确度达到96.7%,目标跟踪准确度提高了9.547%,编号(ID)切换总次数减少了26.4%。

Dual-Path Vision Transformer用于急性缺血性脑卒中辅助诊断

急性缺血性脑卒中是由于脑组织血液供应障碍导致的脑功能障碍,数字减影脑血管造影(DSA)是诊断脑血管疾病的金标准。基于患者的正面和侧面DSA图像,对急性缺血性脑卒中的治疗效果进行分级评估,构建基于Vision Transformer的双路径图像分类智能模型DPVF。为了提高辅助诊断速度,基于EdgeViT的轻量化设计思想进行了模型的构建;为了使模型保持轻量化的同时具有较高的精度,提出空间-通道自注意力模块,促进Transformer模型捕获更全面的特征信息,提高模型的表达能力;此外,对于DPVF的两分支的特征融合,构建交叉注意力模块对两分支输出进行交叉融合,促使模型提取更丰富的特征,从而提高模型表现。实验结果显示DPVF在测试集上的准确率达98.5%,满足实际需求。

SVMD-PE-BP-Transformer短期光伏功率预测

考虑到光伏功率受气象因素变化影响而波动性大难以预测的问题,将逐次变分模态分解SVMD-排列熵PE与BPTransformer相结合,给出了一种组合预测方法,以下简称SPBT模型。在去除非相关因子的基础上,利用SOM聚类方法,对全年光伏数据进行3种类型的分类;针对光伏发电初始时序中所蕴含的重要信息,利用SVMD自适应K值的方法,对其进行分解。再利用PE方法计算各个子序列的熵值,即序列的起伏复杂程度,根据熵的大小,对频率接近的成分进行重构,将其分为两个区间:复杂度低的部分和复杂度高的部分。最后利用BP网络与Transformer分别对其进行预测,并对预测输出进行综合处理。该文以江苏省一光伏电站观测的气象与功率数据为例,通过比较试验验证了该模型的优势,该模型具有较低的预测误差,有助于提高预测精度。

CNN-Transformer结合对比学习的高光谱与LiDAR数据协同分类

针对高光谱图像(hyperspectral images,HSI)与LiDAR数据多模态分类任务中的跨模态信息表达和特征对齐等问题,提出一种基于对比学习CNN-Transformer高光谱和LiDAR数据协同分类网络(Contrastive Learning based CNNTransformer Network,CLCT-Net)。CLCT-Net通过由ConvNeXt V2 Block构成的共有特征提取模块,获得不同模态间的共性特征,解决异构传感器数据之间语义对齐的问题。构建了包含空间-通道分支和光谱上下文分支的双分支HSI编码器,以及结合频域自注意力机制的LiDAR编码器,以获取更丰富的特征表示。利用集成对比学习进行分类,进一步提升多模态数据协同分类的精度。在Houston 2013和Trento数据集上的实验结果表明,相较于其他高光谱图像和Li‐DAR数据分类模型,本文所提模型获得了更高的地物分类精度,分别达到了92.01%和98.90%,实现了跨模态数据特征的深度挖掘和协同提取。

基于Transformer-GRU网络的4D航迹预测

航空器的4D航迹预测作为基于航迹运行(TBO)的关键技术之一具有非常重要的意义。基于Transformer-GRU(T-GRU)网络,提出了一种新的航迹预测方法,结合Adamax优化器实现了4D航迹预测。利用Transformer网络的自注意力机制对输入序列进行建模,通过GRU网络获取时序数据的特征;对原始航迹数据进行重采样插值和中值滤波等预处理,以便消除数据缺失和异常值等对预测的影响;通过E E、E AT、E CT、E A等误差指标对实验结果进行评价,并与其他常用的航迹预测方法进行对比。研究结果表明:与传统深度学习方法相比,基于T-GRU网络的4D航迹预测模型在航迹预测中具有更高的准确性和鲁棒性。

基于改进Detection Transformer的棉花幼苗与杂草检测模型研究

基于深度学习的目标检测技术在棉花幼苗与杂草检测领域已取得一定进展。论文提出了基于改进Detection Transformer的棉花幼苗与杂草检测模型,以提高杂草目标检测的准确率和效率。首先,引入了可变形注意力模块替代原始模型中的Transformer注意力模块,提高模型对特征图目标形变的处理能力。提出新的降噪训练机制,解决了二分图匹配不稳定问题。提出混合查询选择策略,提高解码器对目标类别和位置信息的利用效率。使用Swin Transformer作为网络主干,提高模型特征提取能力。通过对比原网络,论文提出的模型方法在训练过程中表现出更快的收敛速度,并且在准确率方面提高了6.7%。

维生素D治疗全面性发育迟缓患儿的临床疗效研究

背景 除了某些有明确病因的代谢性疾病导致的全面性发育迟缓(GDD),康复治疗是GDD的主要治疗方式;维生素D通过影响神经营养因子在调节神经细胞的发育和分化方面发挥着重要的神经保护作用;但目前关于补充维生素D对GDD患儿临床疗效的研究开展较少。目的 探讨补充不同剂量的维生素D对GDD患儿康复治疗的临床效果。方法 于2020年9月—2022年6月选取在郑州大学第三附属医院康复医学科首次住院就诊的120例GDD患儿为研究对象,采用随机区组化的方法将其分为常规组(38例)、400 U组(37例)和1 200 U组(35例)。常规组仅进行常规康复治疗;400 U组在常规康复治疗的基础上给予口服400 U/d维生素D;1 200 U组在常规康复训练的基础上给予口服1 200 U/d维生素D。收集3组患儿的性别、就诊年龄等基本资料;于入院时(治疗前)及第3个疗程末(治疗后)行血清25羟维生素D[25(OH)D]水平检测和Gesell发育量表评估[评估适应能力、大运动能力、精细运动能力、语言能力和社交能力5个能区的发育商(DQ)];记录发生在患儿住院期间不良事件的次数,并对上述资料进行分析比较。结果 3组患儿性别、居住地、出生季节、分娩方式、就诊年龄、出生体质量、出生胎龄、主要就诊原因比较,差异均无统计学意义(P>0.05)。治疗前,3组患儿25(OH)D水平、Gesell量表各能区DQ值比较,差异均无统计学意义(P>0.05);治疗后,1 200 U组患儿血25(OH)D水平、Gesell量表大运动能力、精细运动能力、语言能力DQ值高于常规组(P<0.05)。第1、2疗程期间,3组患儿不良事件发生率比较,差异无统计学意义(P>0.05);第3疗程期间,1 200 U组患儿不良事件发生率低于常规组及400 U组(P<0.05)。结论 补充1 200 U维生素D对GDD患儿的康复疗效有益,且能减少康复期间不良事件的发生率。

基于Transformer模型的时序数据预测方法综述

时序数据预测(TSF)是指通过分析历史数据的趋势性、季节性等潜在信息,预测未来时间点或时间段的数值和趋势。时序数据由传感器生成,在金融、医疗、能源、交通、气象等众多领域都发挥着重要作用。随着物联网传感器的发展,海量的时序数据难以使用传统的机器学习解决,而Transformer在自然语言处理和计算机视觉等领域的诸多任务表现优秀,学者们利用Transformer模型有效捕获长期依赖关系,使得时序数据预测任务取得了飞速发展。综述了基于Transformer模型的时序数据预测方法,按时间梳理了时序数据预测的发展进程,系统介绍了时序数据预处理过程和方法,介绍了常用的时序预测评价指标和数据集。以算法框架为研究内容系统阐述了基于Transformer的各类模型在TSF任务中的应用方法和工作原理。通过实验对比了各个模型的性能、优点和局限性,并对实验结果展开了分析与讨论。结合Transformer模型在时序数据预测任务中现有工作存在的挑战提出了该方向未来发展趋势。

小数据集上基于语义的局部注意视觉Transformer方法

在小数据集上从零开始训练时,视觉Transformer无法与同规模的卷积神经网络媲美。基于图像的局部注意力方法,可以显著提高ViT的数据效率,但是会丢失距离较远但相关的补丁之间的信息。为了解决上述问题,提出一种双向并行局部注意力视觉Transformer的方法。该方法首先在特征层面上对补丁进行分组,在组内执行局部注意力,以利用特征空间中补丁之间的关系弥补信息丢失。其次,为了有效融合补丁之间的信息,将基于语义的局部注意力和基于图像的局部注意力并行结合起来,通过双向自适应学习来增强ViT模型在小数据上的性能。实验结果表明,该方法在计算量为15.2 GFLOPs和参数量为57.2 M的情况下,分别在CIFAR-10和CIFAR-100数据集上实现了97.93%和85.80%的准确性。相比于其他方法,双向并行局部注意力视觉Transformer在增强局部引导能力的同时,保持了局部注意力所需属性的有效性。