Enhancing Deep Learning Semantics:The Diffusion Sampling and Label-Driven Co-Attention Approach

The advent of self-attention mechanisms within Transformer models has significantly propelled the advancement of deep learning algorithms,yielding outstanding achievements across diverse domains.Nonetheless,self-attention mechanisms falter when applied to datasets with intricate semantic content and extensive dependency structures.In response,this paper introduces a Diffusion Sampling and Label-Driven Co-attention Neural Network(DSLD),which adopts a diffusion sampling method to capture more comprehensive semantic information of the data.Additionally,themodel leverages the joint correlation information of labels and data to introduce the computation of text representation,correcting semantic representationbiases in thedata,andincreasing the accuracyof semantic representation.Ultimately,the model computes the corresponding classification results by synthesizing these rich data semantic representations.Experiments on seven benchmark datasets show that our proposed model achieves competitive results compared to state-of-the-art methods.

引入上下文信息和Attention Gate的GUS-YOLO遥感目标检测算法

目前基于通用YOLO系列的遥感目标检测算法存在并未充分利用图像的全局上下文信息,在特征融合金字塔部分并未充分考虑缩小融合特征之间的语义鸿沟、抑制冗余信息干扰的缺点。在结合YOLO算法优点的基础上提出GUS-YOLO算法,其拥有一个能够充分利用全局上下文信息的骨干网络Global Backbone。除此之外,该算法在融合特征金字塔自顶向下的结构中引入Attention Gate模块,可以突出必要的特征信息,抑制冗余信息。另外,为Attention Gate模块设计了最佳的网络结构,提出了网络的特征融合结构U-Net。最后,为克服ReLU函数可能导致模型梯度不再更新的问题,该算法将Attention Gate模块的激活函数升级为可学习的SMU激活函数,提高模型鲁棒性。在NWPU VHR-10遥感数据集上,该算法相较于YOLOV7算法取得宽松指标mAP^(0.50)1.64个百分点和严格指标mAP^(0.75)9.39个百分点的性能提升。相较于目前主流的七种检测算法,该算法取得较好的检测性能。

基于XGBoost-WOA-BiLSTM-Attention的公共建筑暖通空调能耗预测研究

为在双碳目标下实现节能减排,降低能源成本,提出一种基于BiLSTM的公共建筑暖通空调能耗预测模型。在BiLSTM模型基础上,使用XGBoost算法对输入特征进行选择,剔除冗余特征,得到最佳模型输入特征;然后利用WOA优化算法对添加了Attention机制的BiLSTM模型中的6个超参数进行优化,将得到的最优参数代入BiLSTM-Attention神经网络中进行预测,并与BiLSTM模型、BiLSTM-Attention模型和WOA-BiLSTM-Attention模型进行对比。结果表明,所提出的XGBoost-WOA-BiLSTM-Attention模型的RMSE、MAE、R2分别为0.0106、0.006、0.9991,优于其他模型,且相对于持续模型在均方根误差RMSE上提升了98%,为降低公共建筑暖通空调能耗研究提供了参考。

Image Inpainting Technique Incorporating Edge Prior and Attention Mechanism

Recently,deep learning-based image inpainting methods have made great strides in reconstructing damaged regions.However,these methods often struggle to produce satisfactory results when dealing with missing images with large holes,leading to distortions in the structure and blurring of textures.To address these problems,we combine the advantages of transformers and convolutions to propose an image inpainting method that incorporates edge priors and attention mechanisms.The proposed method aims to improve the results of inpainting large holes in images by enhancing the accuracy of structure restoration and the ability to recover texture details.This method divides the inpainting task into two phases:edge prediction and image inpainting.Specifically,in the edge prediction phase,a transformer architecture is designed to combine axial attention with standard self-attention.This design enhances the extraction capability of global structural features and location awareness.It also balances the complexity of self-attention operations,resulting in accurate prediction of the edge structure in the defective region.In the image inpainting phase,a multi-scale fusion attention module is introduced.This module makes full use of multi-level distant features and enhances local pixel continuity,thereby significantly improving the quality of image inpainting.To evaluate the performance of our method.comparative experiments are conducted on several datasets,including CelebA,Places2,and Facade.Quantitative experiments show that our method outperforms the other mainstream methods.Specifically,it improves Peak Signal-to-Noise Ratio(PSNR)and Structure Similarity Index Measure(SSIM)by 1.141~3.234 db and 0.083~0.235,respectively.Moreover,it reduces Learning Perceptual Image Patch Similarity(LPIPS)and Mean Absolute Error(MAE)by 0.0347~0.1753 and 0.0104~0.0402,respectively.Qualitative experiments reveal that our method excels at reconstructing images with complete structural information and clear texture details.Furthermore,our model exhibits impressive performance in terms of the number of parameters,memory cost,and testing time.

Deep neural network based on multi-level wavelet and attention for structured illumination microscopy

Structured illumination microscopy(SIM)is a popular and powerful super-resolution(SR)technique in biomedical research.However,the conventional reconstruction algorithm for SIM heavily relies on the accurate prior knowledge of illumination patterns and signal-to-noise ratio(SNR)of raw images.To obtain high-quality SR images,several raw images need to be captured under high fluorescence level,which further restricts SIM’s temporal resolution and its applications.Deep learning(DL)is a data-driven technology that has been used to expand the limits of optical microscopy.In this study,we propose a deep neural network based on multi-level wavelet and attention mechanism(MWAM)for SIM.Our results show that the MWAM network can extract high-frequency information contained in SIM raw images and accurately integrate it into the output image,resulting in superior SR images compared to those generated using wide-field images as input data.We also demonstrate that the number of SIM raw images can be reduced to three,with one image in each illumination orientation,to achieve the optimal tradeoff between temporal and spatial resolution.Furthermore,our MWAM network exhibits superior reconstruction ability on low-SNR images compared to conventional SIM algorithms.We have also analyzed the adaptability of this network on other biological samples and successfully applied the pretrained model to other SIM systems.

Attention Mechanism-Based Method for Intrusion Target Recognition in Railway

The detection of foreign object intrusion is crucial for ensuring the safety of railway operations.To address challenges such as low efficiency,suboptimal detection accuracy,and slow detection speed inherent in conventional comprehensive video monitoring systems for railways,a railway foreign object intrusion recognition and detection system is conceived and implemented using edge computing and deep learning technologies.In a bid to raise detection accuracy,the convolutional block attention module(CBAM),including spatial and channel attention modules,is seamlessly integrated into the YOLOv5 model,giving rise to the CBAM-YOLOv5 model.Furthermore,the distance intersection-over-union_non-maximum suppression(DIo U_NMS)algorithm is employed in lieu of the weighted nonmaximum suppression algorithm,resulting in improved detection performance for intrusive targets.To accelerate detection speed,the model undergoes pruning based on the batch normalization(BN)layer,and Tensor RT inference acceleration techniques are employed,culminating in the successful deployment of the algorithm on edge devices.The CBAM-YOLOv5 model exhibits a notable 2.1%enhancement in detection accuracy when evaluated on a selfconstructed railway dataset,achieving 95.0%for mean average precision(m AP).Furthermore,the inference speed on edge devices attains a commendable 15 frame/s.

Main focus of parents of children with attention deficit hyperactivity disorder and the effectiveness of early clinical screening

BACKGROUND Attention deficit hyperactivity disorder(ADHD)is a common mental and behavioral disorder among children.AIM To explore the focus of attention deficit hyperactivity disorder parents and the effectiveness of early clinical screening METHODS This study found that the main directions of parents seeking medical help were short attention time for children under 7 years old(16.6%)and poor academic performance for children over 7 years old(12.1%).We employed a two-stage experiment to diagnose ADHD.Among the 5683 children evaluated from 2018 to 2021,360 met the DSM-5 criteria.Those diagnosed with ADHD underwent assessments for letter,number,and figure attention.Following the exclusion of ADHD-H diagnoses,the detection rate rose to 96.0%,with 310 out of 323 cases identified.RESULTS This study yielded insights into the primary concerns of parents regarding their children's symptoms and validated the efficacy of a straightforward diagnostic test,offering valuable guidance for directing ADHD treatment,facilitating early detection,and enabling timely intervention.Our research delved into the predominant worries of parents across various age groups.Furthermore,we showcased the precision of the simple exclusion experiment in discerning between ADHD-I and ADHD-C in children.CONCLUSION Our study will help diagnose and guide future treatment directions for ADHD.

The Short-Term Prediction ofWind Power Based on the Convolutional Graph Attention Deep Neural Network

The fluctuation of wind power affects the operating safety and power consumption of the electric power grid and restricts the grid connection of wind power on a large scale.Therefore,wind power forecasting plays a key role in improving the safety and economic benefits of the power grid.This paper proposes a wind power predicting method based on a convolutional graph attention deep neural network with multi-wind farm data.Based on the graph attention network and attention mechanism,the method extracts spatial-temporal characteristics from the data of multiple wind farms.Then,combined with a deep neural network,a convolutional graph attention deep neural network model is constructed.Finally,the model is trained with the quantile regression loss function to achieve the wind power deterministic and probabilistic prediction based on multi-wind farm spatial-temporal data.A wind power dataset in the U.S.is taken as an example to demonstrate the efficacy of the proposed model.Compared with the selected baseline methods,the proposed model achieves the best prediction performance.The point prediction errors(i.e.,root mean square error(RMSE)and normalized mean absolute percentage error(NMAPE))are 0.304 MW and 1.177%,respectively.And the comprehensive performance of probabilistic prediction(i.e.,con-tinuously ranked probability score(CRPS))is 0.580.Thus,the significance of multi-wind farm data and spatial-temporal feature extraction module is self-evident.

New Fusion Approach of Spatial and Channel Attention for Semantic Segmentation of Very High Spatial Resolution Remote Sensing Images

The semantic segmentation of very high spatial resolution remote sensing images is difficult due to the complexity of interpreting the interactions between the objects in the scene. Indeed, effective segmentation requires considering spatial local context and long-term dependencies. To address this problem, the proposed approach is inspired by the MAC-UNet network which is an extension of U-Net, densely connected combined with channel attention. The advantages of this solution are as follows: 4) The new model introduces a new attention called propagate attention to build an attention-based encoder. 2) The fusion of multi-scale information is achieved by a weighted linear combination of the attentions whose coefficients are learned during the training phase. 3) Introducing in the decoder, the Spatial-Channel-Global-Local block which is an attention layer that uniquely combines channel attention and spatial attention locally and globally. The performances of the model are evaluated on 2 datasets WHDLD and DLRSD and show results of mean intersection over union (mIoU) index in progress between 1.54% and 10.47% for DLRSD and between 1.04% and 4.37% for WHDLD compared with the most efficient algorithms with attention mechanisms like MAU-Net and transformers like TMNet.

基于Coordinate Attention和空洞卷积的异物识别

在我国工厂的工业化生产中,带式运输机占有重要的地位,但是在其运输物料的过程中,常有木板、金属管、大型金属片等混入物料中,从而对带式运输机的传送带造成损毁,引起巨大的经济损失.为了检测出传送带上的不规则异物,设计了一种新的异物检测方法.针对传统异物检测方法中存在的对于图像特征提取能力不足以及网络感受野相对较小的问题,我们提出了一种基于coordinate attention和空洞卷积的单阶段异物识别方法.首先,网络利用coordinate attention机制,使网络更加关注图像的空间信息,并对图像中的重要特征进行了增强,增强了网络的性能;其次,在网络提取多尺度特征的部分,将原网络的静态卷积变为空洞卷积,有效减少了常规卷积造成的信息损失;除此之外,我们还使用了新的损失函数,进一步提高了网络的性能.实验结果证明,我们提出的网络能有效识别出传送带上的异物,较好地完成异物检测任务.

基于ALBERT-Seq2Seq-Attention模型的数字化档案多标签分类

针对现有的数字化档案多标签分类方法存在分类标签之间缺少关联性的问题,提出一种用于档案多标签分类的深层神经网络模型ALBERT-Seq2Seq-Attention.该模型通过ALBERT(A Little BERT)预训练语言模型内部多层双向的Transfomer结构获取进行文本特征向量的提取,并获得上下文语义信息;将预训练提取的文本特征作为Seq2Seq-Attention(Sequence to Sequence-Attention)模型的输入序列,构建标签字典以获取多标签间的关联关系.将分类模型在3种数据集上分别进行对比实验,结果表明:模型分类的效果F1值均超过90%.该模型不仅能提高档案文本的多标签分类效果,也能关注标签之间的相关关系.

Attention Markets of Blockchain-Based Decentralized Autonomous Organizations

The attention is a scarce resource in decentralized autonomous organizations(DAOs),as their self-governance relies heavily on the attention-intensive decision-making process of“proposal and voting”.To prevent the negative effects of pro-posers’attention-capturing strategies that contribute to the“tragedy of the commons”and ensure an efficient distribution of attention among multiple proposals,it is necessary to establish a market-driven allocation scheme for DAOs’attention.First,the Harberger tax-based attention markets are designed to facilitate its allocation via continuous and automated trading,where the individualized Harberger tax rate(HTR)determined by the pro-posers’reputation is adopted.Then,the Stackelberg game model is formulated in these markets,casting attention to owners in the role of leaders and other competitive proposers as followers.Its equilibrium trading strategies are also discussed to unravel the intricate dynamics of attention pricing.Moreover,utilizing the single-round Stackelberg game as an illustrative example,the existence of Nash equilibrium trading strategies is demonstrated.Finally,the impact of individualized HTR on trading strategies is investigated,and results suggest that it has a negative correlation with leaders’self-accessed prices and ownership duration,but its effect on their revenues varies under different conditions.This study is expected to provide valuable insights into leveraging attention resources to improve DAOs’governance and decision-making process.

基于CNN-LSTM-Attention和自回归的混合水位预测模型

【目的】水位预测对交通运输、农业以及防洪措施具有重要影响。精确的水位值可用于提升水道运输的安全及效率、降低洪水风险,同时也是保障区域可持续发展的必要条件。【方法】提出一种CRANet的混合水位预测模型,以卷积神经网络(CNN)、长短期记忆网络(LSTM)、注意力机制以及自回归(AR)组件为基础,旨在应对时间序列数据中存在的线性与非线性问题,缓解自回归及ARIMA模型的缺陷。其应用不仅在于为航运调度提供决策支撑,加强导航安全效率,同样能提升防洪减灾的能力。其中,CNN和LSTM组件有效地针对数据集内的局部和全局关系进行捕捉,AR组件则能充分考虑数据的时间序列特性。同时,通过注意力机制,模型能够优先考虑相关特性,提高预测效果。【结果】研究成果所提出的模型已成功应用于中国西江梧州站的水位预测,在测试集上预测未来3 h级别水位的MAE、RMSE和R^(2)分别为0.086、0.114 5和0.950 8。【结论】结果表明所提出的CRANet模型在水位预测方面的高可用性、准确度与稳健性,相较于AR、SVR、CNN、LSTM等模型具有更优的MAE、RMSE和R^(2)。

An Underwater Target Detection Algorithm Based on Attention Mechanism and Improved YOLOv7

For underwater robots in the process of performing target detection tasks,the color distortion and the uneven quality of underwater images lead to great difficulties in the feature extraction process of the model,which is prone to issues like error detection,omission detection,and poor accuracy.Therefore,this paper proposed the CER-YOLOv7(CBAM-EIOU-RepVGG-YOLOv7)underwater target detection algorithm.To improve the algorithm’s capability to retain valid features from both spatial and channel perspectives during the feature extraction phase,we have added a Convolutional Block Attention Module(CBAM)to the backbone network.The Reparameterization Visual Geometry Group(RepVGG)module is inserted into the backbone to improve the training and inference capabilities.The Efficient Intersection over Union(EIoU)loss is also used as the localization loss function,which reduces the error detection rate and missed detection rate of the algorithm.The experimental results of the CER-YOLOv7 algorithm on the UPRC(Underwater Robot Prototype Competition)dataset show that the mAP(mean Average Precision)score of the algorithm is 86.1%,which is a 2.2%improvement compared to the YOLOv7.The feasibility and validity of the CER-YOLOv7 are proved through ablation and comparison experiments,and it is more suitable for underwater target detection.

Efficient Unsupervised Image Stitching Using Attention Mechanism with Deep Homography Estimation

Traditional feature-based image stitching techniques often encounter obstacles when dealing with images lackingunique attributes or suffering from quality degradation. The scarcity of annotated datasets in real-life scenesseverely undermines the reliability of supervised learning methods in image stitching. Furthermore, existing deeplearning architectures designed for image stitching are often too bulky to be deployed on mobile and peripheralcomputing devices. To address these challenges, this study proposes a novel unsupervised image stitching methodbased on the YOLOv8 (You Only Look Once version 8) framework that introduces deep homography networksand attentionmechanisms. Themethodology is partitioned into three distinct stages. The initial stage combines theattention mechanism with a pooling pyramid model to enhance the detection and recognition of compact objectsin images, the task of the deep homography networks module is to estimate the global homography of the inputimages consideringmultiple viewpoints. The second stage involves preliminary stitching of the masks generated inthe initial stage and further enhancement through weighted computation to eliminate common stitching artifacts.The final stage is characterized by adaptive reconstruction and careful refinement of the initial stitching results.Comprehensive experiments acrossmultiple datasets are executed tometiculously assess the proposed model. Ourmethod’s Peak Signal-to-Noise Ratio (PSNR) and Structure Similarity Index Measure (SSIM) improved by 10.6%and 6%. These experimental results confirm the efficacy and utility of the presented model in this paper.

融合MacBERT和Talking⁃Heads Attention实体关系联合抽取模型

针对现有的医学文本关系抽取任务模型在训练过程中存在语义理解能力不足,可能导致关系抽取的效果不尽人意的问题,文中提出一种融合MacBERT和Talking⁃Heads Attention的实体关系联合抽取模型。该模型首先利用MacBERT语言模型来获取动态字向量表达,MacBERT作为改进的BERT模型,能够减少预训练和微调阶段之间的差异,从而提高模型的泛化能力;然后,将这些动态字向量表达输入到双向门控循环单元(BiGRU)中,以便提取文本的上下文特征。BiGRU是一种改进的循环神经网络(RNN),具有更好的长期依赖捕获能力。在获取文本上下文特征之后,使用Talking⁃Heads Attention来获取全局特征。Talking⁃Heads Attention是一种自注意力机制,可以捕获文本中不同位置之间的关系,从而提高关系抽取的准确性。实验结果表明,与实体关系联合抽取模型GRTE相比,该模型F1值提升1%,precision值提升0.4%,recall值提升1.5%。

基于Attention机制和递归思想的LSTM车辆轨迹预测

针对现有车辆轨迹预测模型在长时预测方面准确性不足的问题,基于Attention机制和递归思想的长短时记忆网络(long short-term memory, LSTM)构建了一种新型的车辆轨迹预测模型,即ATT-LSTM(RE)模型,使用编码器–解码器架构更精确地预测车辆未来的行驶轨迹。研究结果表明,模型意图识别的准确率为91.7%,F1分数、召回率、精确率均在0.872~0.977之间;1 s、2 s、3 s、4 s、5 s的终点轨迹预测的均方根误差为0.52 m、1.07 m、1.69 m、2.58 m、3.31 m,优于同类型模型。

基于双向LSTM-Attention模型的火电厂负荷预测研究

准确预测电厂负荷可指导火电厂制定发电计划和调度安排,有利于降低能源成本和污染物排放,对电厂的经济性和环保性有重要意义。本文提出一种基于双向LSTM-Attention的火电厂负荷预测方法。首先,通过皮尔逊系数筛选出关键特征变量;其次利用双向长短期记忆网络提取关键变量之间的长期依赖关系与短期变化特征,最后融合注意力权重机制以进一步突出关键时序信息,进而实现负荷的准确预测。以某在役600 MW超临界机组为对象进行验证。结果表明:相较于单向LSTM、双向LSTM、单向LSTM-Attention,本文所提方法的决定系数R^(2)、均方根误差S_(RMSE)和平均绝对误差S_(MAE)均为最优,分别为0.9566、16.3159、13.5043,能更准确地捕捉到负荷快速波动的趋势,为电厂的负荷预测和能源管理提供可行的方法。

基于Attention-BiLSTM混合模型的月尺度降水量预测

降水受到多种气象因素的影响,从而导致降水预测精度不高。针对这个问题,在考虑影响降水的多个气象因素基础上,通过Attention机制赋予各种气象因素不同的权重,结合双向长短期记忆神经网络(BiLSTM),提出了改进的Attention-BiLSTM混合模型去实现月尺度降水量的预测。以江西省南昌气象站为例,将1989—2018年的逐月降水量与逐月气象因素(气温、蒸发量、气压等)观测资料作为模型输入数据,通过Attention机制识别出各种气象因素的权重,从而提高BiLSTM模型对降水量的预测性能。结果表明:Attention-BiLSTM混合模型可有效地提高降水量预测的精度;通过Attention机制的修正,显著地改善了原有的BiLSTM模型降水量预测值偏低的问题。

Causal temporal graph attention network for fault diagnosis of chemical processes

Fault detection and diagnosis(FDD)plays a significant role in ensuring the safety and stability of chemical processes.With the development of artificial intelligence(AI)and big data technologies,data-driven approaches with excellent performance are widely used for FDD in chemical processes.However,improved predictive accuracy has often been achieved through increased model complexity,which turns models into black-box methods and causes uncertainty regarding their decisions.In this study,a causal temporal graph attention network(CTGAN)is proposed for fault diagnosis of chemical processes.A chemical causal graph is built by causal inference to represent the propagation path of faults.The attention mechanism and chemical causal graph were combined to help us notice the key variables relating to fault fluctuations.Experiments in the Tennessee Eastman(TE)process and the green ammonia(GA)process showed that CTGAN achieved high performance and good explainability.