Classification of Conversational Sentences Using an Ensemble Pre-Trained Language Model with the Fine-Tuned Parameter

Sentence classification is the process of categorizing a sentence based on the context of the sentence.Sentence categorization requires more semantic highlights than other tasks,such as dependence parsing,which requires more syntactic elements.Most existing strategies focus on the general semantics of a conversation without involving the context of the sentence,recognizing the progress and comparing impacts.An ensemble pre-trained language model was taken up here to classify the conversation sentences from the conversation corpus.The conversational sentences are classified into four categories:information,question,directive,and commission.These classification label sequences are for analyzing the conversation progress and predicting the pecking order of the conversation.Ensemble of Bidirectional Encoder for Representation of Transformer(BERT),Robustly Optimized BERT pretraining Approach(RoBERTa),Generative Pre-Trained Transformer(GPT),DistilBERT and Generalized Autoregressive Pretraining for Language Understanding(XLNet)models are trained on conversation corpus with hyperparameters.Hyperparameter tuning approach is carried out for better performance on sentence classification.This Ensemble of Pre-trained Language Models with a Hyperparameter Tuning(EPLM-HT)system is trained on an annotated conversation dataset.The proposed approach outperformed compared to the base BERT,GPT,DistilBERT and XLNet transformer models.The proposed ensemble model with the fine-tuned parameters achieved an F1_score of 0.88.

New approach to assess sperm DNA fragmentation dynamics： Fine-tuning mathematical models

Background： Sperm DNA fragmentation（sDF） has been proved to be an important parameter in order to predict in vitro the potential fertility of a semen sample. Colloid centrifugation could be a suitable technique to select those donkey sperm more resistant to DNA fragmentation after thawing. Previous studies have shown that to elucidate the latent damage of the DNA molecule, sDF should be assessed dynamically, where the rate of fragmentation between treatments indicates how resistant the DNA is to iatrogenic damage. The rate of fragmentation is calculated using the slope of a linear regression equation. However, it has not been studied if s DF dynamics fit this model. The objectives of this study were to evaluate the effect of different after-thawing centrifugation protocols on sperm DNA fragmentation and elucidate the most accurate mathematical model（linear regression, exponential or polynomial） for DNA fragmentation over time in frozen-thawed donkey semen.Results： After submitting post-thaw semen samples to no centrifugation（UDC）, sperm washing（SW） or single layer centrifugation（SLC） protocols, sD F values after 6 h of incubation were significantly lower in SLC samples than in SW or UDC.Coefficient of determination（R-2） values were significantly higher for a second order polynomial model than for linear or exponential. The highest values for acceleration of fragmentation（aSDF） were obtained for SW, fol owed by SLC and UDC.Conclusion： SLC after thawing seems to preserve longer DNA longevity in comparison to UDC and SW. Moreover,the fine-tuning of models has shown that sDF dynamics in frozen-thawed donkey semen fit a second order polynomial model, which implies that fragmentation rate is not constant and fragmentation acceleration must be taken into account to elucidate hidden damage in the DNA molecule.

Rotary-scaling fine-tuning (RSFT) method for optimizing railway wheel profiles and its application to a locomotive

The existing multi-objective wheel profile optimization methods mainly consist of three sub-modules:(1)wheel profile generation,(2)multi-body dynamics simulation,and(3)an optimization algorithm.For the first module,a comparably conservative rotary-scaling finetuning(RSFT)method,which introduces two design variables and an empirical formula,is proposed to fine-tune the traditional wheel profiles for improving their engineering applicability.For the second module,for the TRAXX locomotives serving on the Blankenburg–Rubeland line,an optimization function representing the relationship between the wheel profile and the wheel–rail wear number is established based on Kriging surrogate model(KSM).For the third module,a method combining the regression capability of KSM with the iterative computing power of particle swarm optimization(PSO)is proposed to quickly and reliably implement the task of optimizing wheel profiles.Finally,with the RSFT–KSM–PSO method,we propose two wear-resistant wheel profiles for the TRAXX locomotives serving on the Blankenburg–Rubeland line,namely S1002-S and S1002-M.The S1002-S profile minimizes the total wear number by 30%,while the S1002-M profile makes the wear distribution more uniform through a proper sacrifice of the tread wear number,and the total wear number is reduced by 21%.The quasi-static and hunting stability tests further demonstrate that the profile designed by the RSFT–KSM–PSO method is promising for practical engineering applications.

Joint On-Demand Pruning and Online Distillation in Automatic Speech Recognition Language Model Optimization

Automatic speech recognition(ASR)systems have emerged as indispensable tools across a wide spectrum of applications,ranging from transcription services to voice-activated assistants.To enhance the performance of these systems,it is important to deploy efficient models capable of adapting to diverse deployment conditions.In recent years,on-demand pruning methods have obtained significant attention within the ASR domain due to their adaptability in various deployment scenarios.However,these methods often confront substantial trade-offs,particularly in terms of unstable accuracy when reducing the model size.To address challenges,this study introduces two crucial empirical findings.Firstly,it proposes the incorporation of an online distillation mechanism during on-demand pruning training,which holds the promise of maintaining more consistent accuracy levels.Secondly,it proposes the utilization of the Mogrifier long short-term memory(LSTM)language model(LM),an advanced iteration of the conventional LSTM LM,as an effective alternative for pruning targets within the ASR framework.Through rigorous experimentation on the ASR system,employing the Mogrifier LSTM LM and training it using the suggested joint on-demand pruning and online distillation method,this study provides compelling evidence.The results exhibit that the proposed methods significantly outperform a benchmark model trained solely with on-demand pruning methods.Impressively,the proposed strategic configuration successfully reduces the parameter count by approximately 39%,all the while minimizing trade-offs.

PAL-BERT:An Improved Question Answering Model

In the field of natural language processing(NLP),there have been various pre-training language models in recent years,with question answering systems gaining significant attention.However,as algorithms,data,and computing power advance,the issue of increasingly larger models and a growing number of parameters has surfaced.Consequently,model training has become more costly and less efficient.To enhance the efficiency and accuracy of the training process while reducing themodel volume,this paper proposes a first-order pruningmodel PAL-BERT based on the ALBERT model according to the characteristics of question-answering(QA)system and language model.Firstly,a first-order network pruning method based on the ALBERT model is designed,and the PAL-BERT model is formed.Then,the parameter optimization strategy of the PAL-BERT model is formulated,and the Mish function was used as an activation function instead of ReLU to improve the performance.Finally,after comparison experiments with traditional deep learning models TextCNN and BiLSTM,it is confirmed that PALBERT is a pruning model compression method that can significantly reduce training time and optimize training efficiency.Compared with traditional models,PAL-BERT significantly improves the NLP task’s performance.

Research status and application of artificial intelligence large models in the oil and gas industry

This article elucidates the concept of large model technology,summarizes the research status of large model technology both domestically and internationally,provides an overview of the application status of large models in vertical industries,outlines the challenges and issues confronted in applying large models in the oil and gas sector,and offers prospects for the application of large models in the oil and gas industry.The existing large models can be briefly divided into three categories:large language models,visual large models,and multimodal large models.The application of large models in the oil and gas industry is still in its infancy.Based on open-source large language models,some oil and gas enterprises have released large language model products using methods like fine-tuning and retrieval augmented generation.Scholars have attempted to develop scenario-specific models for oil and gas operations by using visual/multimodal foundation models.A few researchers have constructed pre-trained foundation models for seismic data processing and interpretation,as well as core analysis.The application of large models in the oil and gas industry faces challenges such as current data quantity and quality being difficult to support the training of large models,high research and development costs,and poor algorithm autonomy and control.The application of large models should be guided by the needs of oil and gas business,taking the application of large models as an opportunity to improve data lifecycle management,enhance data governance capabilities,promote the construction of computing power,strengthen the construction of“artificial intelligence+energy”composite teams,and boost the autonomy and control of large model technology.

Construction and preliminary application of large language model for reservoir performance analysis

A large language model(LLM)is constructed to address the sophisticated demands of data retrieval and analysis,detailed well profiling,computation of key technical indicators,and the solutions to complex problems in reservoir performance analysis(RPA).The LLM is constructed for RPA scenarios with incremental pre-training,fine-tuning,and functional subsystems coupling.Functional subsystem and efficient coupling methods are proposed based on named entity recognition(NER),tool invocation,and Text-to-SQL construction,all aimed at resolving pivotal challenges in developing the specific application of LLMs for RDA.This study conducted a detailed accuracy test on feature extraction models,tool classification models,data retrieval models and analysis recommendation models.The results indicate that these models have demonstrated good performance in various key aspects of reservoir dynamic analysis.The research takes some injection and production well groups in the PK3 Block of the Daqing Oilfield as an example for testing.Testing results show that our model has significant potential and practical value in assisting reservoir engineers with RDA.The research results provide a powerful support to the application of LLM in reservoir performance analysis.

Optimizing Fine-Tuning in Quantized Language Models:An In-Depth Analysis of Key Variables

Large-scale Language Models(LLMs)have achieved significant breakthroughs in Natural Language Processing(NLP),driven by the pre-training and fine-tuning paradigm.While this approach allows models to specialize in specific tasks with reduced training costs,the substantial memory requirements during fine-tuning present a barrier to broader deployment.Parameter-Efficient Fine-Tuning(PEFT)techniques,such as Low-Rank Adaptation(LoRA),and parameter quantization methods have emerged as solutions to address these challenges by optimizing memory usage and computational efficiency.Among these,QLoRA,which combines PEFT and quantization,has demonstrated notable success in reducing memory footprints during fine-tuning,prompting the development of various QLoRA variants.Despite these advancements,the quantitative impact of key variables on the fine-tuning performance of quantized LLMs remains underexplored.This study presents a comprehensive analysis of these key variables,focusing on their influence across different layer types and depths within LLM architectures.Our investigation uncovers several critical findings:(1)Larger layers,such as MLP layers,can maintain performance despite reductions in adapter rank,while smaller layers,like self-attention layers,aremore sensitive to such changes;(2)The effectiveness of balancing factors depends more on specific values rather than layer type or depth;(3)In quantization-aware fine-tuning,larger layers can effectively utilize smaller adapters,whereas smaller layers struggle to do so.These insights suggest that layer type is a more significant determinant of fine-tuning success than layer depth when optimizing quantized LLMs.Moreover,for the same discount of trainable parameters,reducing the trainable parameters in a larger layer is more effective in preserving fine-tuning accuracy than in a smaller one.This study provides valuable guidance for more efficient fine-tuning strategies and opens avenues for further research into optimizing LLM fine-tuning in resource-constrained environments.

Training and Implementation of Subjective Questions Scoring System Based on the Baidu Qianfan Model Platform

Leveraging the Baidu Qianfan model platform,this paper designs and implements a highly efficient and accurate scoring system for subjective questions,focusing primarily on questions in the field of computer network technology.The system enhances the foundational model by utilizing Qianfan’s training tools and integrating advanced techniques,such as supervised fine-tuning.In the data preparation phase,a comprehensive collection of subjective data related to computer network technology is gathered,cleaned,and labeled.During model training and evaluation,optimal hyperparameters and tuning strategies are applied,resulting in a model capable of scoring with high accuracy.Evaluation results demonstrate that the proposed model performs well across multiple dimensions-content,expression,and development scores-yielding results comparable to those of manual scoring.

A Novel Deep Neural Network Compression Model for Airport Object Detection

A novel deep neural network compression model for airport object detection has been presented.This novel model aims at disadvantages of deep neural network,i.e.the complexity of the model and the great cost of calculation.According to the requirement of airport object detection,the model obtains temporal and spatial semantic rules from the uncompressed model.These spatial semantic rules are added to the model after parameter compression to assist the detection.The rules can improve the accuracy of the detection model in order to make up for the loss caused by parameter compression.The experiments show that the effect of the novel compression detection model is no worse than that of the uncompressed original model.Even some of the original model false detection can be eliminated through the prior knowledge.

改进YOLOv8的轻量级军事飞机检测算法

遥感图像军事飞机检测在侦察预警、情报分析等领域具有重要意义。为使军事飞机检测模型能在算力受限的设备上高效运行,从网络设计与模型压缩两个方面对YOLOv8n进行轻量化改进。在网络设计方面,使用FAS_C2f替换原始主干网络中的C2f模块,减少计算冗余并加快网络特征提取的速度;根据军事飞机目标的尺度特征对网络结构进行优化,缓解因过度下采样导致的小目标信息丢失问题;使用Inner-SIoU作为新的定位回归损失函数,提升对小目标样本的学习能力并加快回归边界框的收敛。在模型压缩方面,使用基于LAMP分数的通道剪枝对重设计后的模型进行压缩,进一步减少参数和模型大小;并利用通道级知识蒸馏(channel-wise knowledge distillation,CWD)将模型精度恢复到接近剪枝前的水平。实验结果表明,在公开军用飞机数据集MAR20上,轻量化后的模型mAP为97.2%,体积仅有0.7 MB,较原始模型缩小了88.3%,FPS提高了14帧/s,满足军事飞机目标检测的实时性要求。

Multi-grained Pruning Method of Convolutional Neural Network

Although the deep learning technology has shown great power in solving the complex tasks, these neural network models are large and redundant as a matter of fact, which makes these networks difficult to be placed in embedded devices with limited memory and computing resources. In order to compress the neural network to a slimmer and smaller one, the multi-grained network pruning framework is proposed in this paper. In our framework, the pruning process was divided into the filter-level pruning and the weight-level pruning. In the process of the filter-level pruning, the importance of the filter was measured by the entropy of the activation tensor of the filter. In the other process, the dynamic recoverable pruning method was adopted to prune the weights deeply. Different from these popular pruning methods, the weight-level pruning is also taken into account based on the employment of the filter-level pruning to achieve more effectively pruning. The proposed approach is validated on two representative CNN models - AlexNet and VGG16, pre-trained on ILSVRC12. Experimental results show that AlexNet and VGG16 network models are compressed 19.75- and 22.53- respectively by this approach, which are 2.05 and 5.89 higher than the classical approaches of dynamic Network Surgery and ThiNet.

注意力引导的多尺度红外行人车辆实时检测

红外成像技术通过捕捉目标热辐射特征进行成像,能实现复杂道路场景下的目标监测和道路冗杂信息滤除。针对红外行人和车辆目标检测模型参数量大、依赖高性能GPU资源和检测速度慢等问题,提出了一种注意力引导的多尺度红外行人车辆实时检测模型。首先,为精确匹配校准红外行人和车辆目标尺度与锚框尺寸,利用K-Means++算法对红外行人和车辆目标尺度进行先验框预置参数重聚类生成,并设计128×128精细尺度检测层;其次,设计注意力引导广域特征提取模块增强模型特征提取能力和空间及通道信息聚焦能力;随后,构建跨空间感知模块引入空间信息感知,强化不同尺度空间下的目标的特征表达能力;最后,针对资源受限设备,通过4倍通道剪枝方法降低模型参数量,增强移动端算法部署适应性。实验结果表明:所提IRDet算法与基准方法相比,模型平均检测精度提升4.3%,达到87.4%,模型权重值压缩60.4%,降至5.7 MB。

神经网络压缩联合优化方法的研究综述

随着人工智能应用的实时性、隐私性和安全性需求增大,在边缘计算平台上部署高性能的神经网络成为研究热点。由于常见的边缘计算平台在存储、算力、功耗上均存在限制,因此深度神经网络的端侧部署仍然是一个巨大的挑战。目前,克服上述挑战的一个思路是对现有的神经网络压缩以适配设备部署条件。现阶段常用的模型压缩算法有剪枝、量化、知识蒸馏,多种方法优势互补同时联合压缩可实现更好的压缩加速效果,正成为研究的热点。本文首先对常用的模型压缩算法进行简要概述,然后总结了“知识蒸馏+剪枝”、“知识蒸馏+量化”和“剪枝+量化”3种常见的联合压缩算法,重点分析论述了联合压缩的基本思想和方法,最后提出了神经网络压缩联合优化方法未来的重点发展方向。

视觉深度学习模型压缩加速综述

近年来,深度学习模型规模越来越大,在嵌入式设备等资源受限环境中,大规模视觉深度学习模型难以实现高效推理部署。模型压缩加速可以有效解决该挑战。尽管已经出现相关工作的综述,但相关工作集中在卷积神经网络的压缩加速,缺乏对视觉Transformer模型压缩加速方法的整理和对比分析。因此,本文以视觉深度学习模型压缩技术为核心,对卷积神经网络和视觉Transformer模型2个最重要的视觉深度模型进行了相关技术手段的整理,并对技术热点和挑战进行了总结和分析。本文旨在为研究者提供一个全面了解模型压缩和加速领域的视角,促进深度学习模型压缩加速技术的发展。

自动语音识别模型压缩算法综述

随着深度学习技术的发展,自动语音识别任务模型的参数数量越来越庞大,使得模型的计算开销、存储需求和功耗花费逐渐增加,难以在资源受限设备上部署.因此对基于深度学习的自动语音识别模型进行压缩,在降低模型大小的同时尽量保持原有性能具有重要价值.针对上述问题,全面综述了近年来该领域的主要工作,将其归纳为知识蒸馏、模型量化、低秩分解、网络剪枝、参数共享以及组合模型几类方法,并进行了系统综述,为模型在资源受限设备的部署提供可选的解决方案.

改进DenseNet模型在工件表面粗糙度视觉检测中的应用

针对原始DenseNet模型检测工件表面粗糙度时间长、准确率较低的问题,结合卷积层滤波器注意力机制和批归一化层缩放系数提出一种工件表面粗糙度检测的深度学习模型。首先,利用注意力重要性值判定模块内的冗余通道。其次,在Dense Block模块内引入批归一化层缩放系数判别特征通道的重要程度。最后联合卷积层滤波器的注意力重要性值和批归一化层缩放系数裁剪冗余通道,实现模型剪枝。实验结果表明,原始DenseNet模型检测工件表面粗糙度的准确率为91.875%,检测时间为483 s。当剪枝率为20%时,其检测效果最好,检测准确率为96.875%,检测时间为255 s。相比于原始DenseNet模型,改进后的DenseNet模型检测效果更好,在质量检测领域方面具有一定的应用前景。

低空轻量级红外弱小目标检测算法

精准的红外弱小目标检测是实时监控、追踪、制导的关键;红外弱小目标存在检测难度高、误检高、漏检严重的问题。为了提高红外弱小目标检测算法的实时性和检测精度,提出了一种超轻量红外弱小目标检测算法SL-YOLO。首先,重设计下采样方案,针对红外图像特征信息调节网络架构,解决红外弱小目标特征梯度降低和特征消失问题;然后设计网络模型剪枝算法,实现剪枝算法与网络结构的融合,去除冗余参数,实现检测速度的提高;最后设计Varifocal-SIoU损失函数,在均衡正负样本与重叠损失的同时,对正样本进行加权处理,解决背景干扰问题。实验结果表明,在SIRST和IDSAT数据集下检测精度分别提高至96.4%、98.1%,模型体积和计算量可压缩至190 kB、0.9 GFLOPs,推理速度降至3 ms以下。与主流算法进行对比,改进后算法在检测精度、模型体积、计算量等方面均取得了不错的成绩。能够满足实时性检测需求。

一种轻量型果园环境果实检测方法

果园环境下柑橘的快速准确检测是自主采摘机器人作业的关键.针对现有的模型过于冗余、检测速度与精度不平衡等问题,提出一种轻量型果园环境果实检测方法.在YOLOv4算法的基础上引入焦点损失函数(Focal Loss)来提高模型在二分类检测任务中的负样本挖掘能力,并针对模型参数冗余等问题提出一种优化的模型剪枝方法.试验结果表明:提出的方法在果园环境中柑橘果实数据集检测得到的平均精度均值(mean average precision,M_(AP))达到94.22%,相较于YOLOv4模型提高了1.18%,模型参数减小了95.22%,模型尺寸为原来的4.84%,检测速度为原来的4.03倍.

一种基于TVM的算子生成加速策略

随着人工智能(AI)的飞速发展,新算子和底层硬件层出不穷,这给算子库的开发和维护带来了巨大的工作量。单纯基于手工优化来解决AI模型的性能和效率很容易出现瓶颈。TVM深度学习编译器通过代码的自动化生成减轻了手工优化的负担,但同时也存在搜索时间长的问题。为此,针对TVM的自动化代码生成框架Ansor,提出基于梯度提升算法的新代价模型和基于预定义规则的调度空间剪枝优化2种优化策略,旨在加速TVM的自动化代码生成过程,实现模型快速落地与部署,并进一步为人工智能技术的应用提供更高效的解决方案。实验结果表明,通过应用优化后代价模型可以在不损失推理时间的前提下,使得在x86 CPU平台上模型的调优时间减少30%~35%,同时优化后算子性能最高可提升22%,使得在深度计算单元(DCU)平台上模型的调优时间减少20%左右,同时优化后算子平均性能提升5.7%,此外,基于预定义规则的剪枝策略可以有效提升代价模型的收敛速度,并且在原有最佳迭代次数下,模型推理时间可提高7.4%。