Convolution Theorem of Fractional Fourier Transformation Derived by Representation Transformation in Quantum Mechancis

Based on our previous paper (Commun.Theor.Phys.39 (2003) 417) we derive the convolution theoremof fractional Fourier transformation in the context of quantum mechanics,which seems a convenient and neat way.Generalization of this method to the complex fractional Fourier transformation case is also possible.

BSTFNet:An Encrypted Malicious Traffic Classification Method Integrating Global Semantic and Spatiotemporal Features

While encryption technology safeguards the security of network communications,malicious traffic also uses encryption protocols to obscure its malicious behavior.To address the issues of traditional machine learning methods relying on expert experience and the insufficient representation capabilities of existing deep learning methods for encrypted malicious traffic,we propose an encrypted malicious traffic classification method that integrates global semantic features with local spatiotemporal features,called BERT-based Spatio-Temporal Features Network(BSTFNet).At the packet-level granularity,the model captures the global semantic features of packets through the attention mechanism of the Bidirectional Encoder Representations from Transformers(BERT)model.At the byte-level granularity,we initially employ the Bidirectional Gated Recurrent Unit(BiGRU)model to extract temporal features from bytes,followed by the utilization of the Text Convolutional Neural Network(TextCNN)model with multi-sized convolution kernels to extract local multi-receptive field spatial features.The fusion of features from both granularities serves as the ultimate multidimensional representation of malicious traffic.Our approach achieves accuracy and F1-score of 99.39%and 99.40%,respectively,on the publicly available USTC-TFC2016 dataset,and effectively reduces sample confusion within the Neris and Virut categories.The experimental results demonstrate that our method has outstanding representation and classification capabilities for encrypted malicious traffic.

基于BERT与细粒度特征提取的数据法学问答系统

首先利用bidirectional encoder representations from transformers(BERT)模型的强大的语境理解能力来提取数据法律文本的深层语义特征,然后引入细粒度特征提取层,依照注意力机制,重点关注文本中与数据法律问答相关的关键部分,最后对所采集的法律问答数据集进行训练和评估.结果显示:与传统的多个单一模型相比,所提出的模型在准确度、精确度、召回率、F1分数等关键性能指标上均有提升,表明该系统能够更有效地理解和回应复杂的数据法学问题,为研究数据法学的专业人士和公众用户提供更高质量的问答服务.

Enhanced Topic-Aware Summarization Using Statistical Graph Neural Networks

The rapid expansion of online content and big data has precipitated an urgent need for efficient summarization techniques to swiftly comprehend vast textual documents without compromising their original integrity.Current approaches in Extractive Text Summarization(ETS)leverage the modeling of inter-sentence relationships,a task of paramount importance in producing coherent summaries.This study introduces an innovative model that integrates Graph Attention Networks(GATs)with Transformer-based Bidirectional Encoder Representa-tions from Transformers(BERT)and Latent Dirichlet Allocation(LDA),further enhanced by Term Frequency-Inverse Document Frequency(TF-IDF)values,to improve sentence selection by capturing comprehensive topical information.Our approach constructs a graph with nodes representing sentences,words,and topics,thereby elevating the interconnectivity and enabling a more refined understanding of text structures.This model is stretched to Multi-Document Summarization(MDS)from Single-Document Summarization,offering significant improvements over existing models such as THGS-GMM and Topic-GraphSum,as demonstrated by empirical evaluations on benchmark news datasets like Cable News Network(CNN)/Daily Mail(DM)and Multi-News.The results consistently demonstrate superior performance,showcasing the model’s robustness in handling complex summarization tasks across single and multi-document contexts.This research not only advances the integration of BERT and LDA within a GATs but also emphasizes our model’s capacity to effectively manage global information and adapt to diverse summarization challenges.

Classification of Conversational Sentences Using an Ensemble Pre-Trained Language Model with the Fine-Tuned Parameter

Sentence classification is the process of categorizing a sentence based on the context of the sentence.Sentence categorization requires more semantic highlights than other tasks,such as dependence parsing,which requires more syntactic elements.Most existing strategies focus on the general semantics of a conversation without involving the context of the sentence,recognizing the progress and comparing impacts.An ensemble pre-trained language model was taken up here to classify the conversation sentences from the conversation corpus.The conversational sentences are classified into four categories:information,question,directive,and commission.These classification label sequences are for analyzing the conversation progress and predicting the pecking order of the conversation.Ensemble of Bidirectional Encoder for Representation of Transformer(BERT),Robustly Optimized BERT pretraining Approach(RoBERTa),Generative Pre-Trained Transformer(GPT),DistilBERT and Generalized Autoregressive Pretraining for Language Understanding(XLNet)models are trained on conversation corpus with hyperparameters.Hyperparameter tuning approach is carried out for better performance on sentence classification.This Ensemble of Pre-trained Language Models with a Hyperparameter Tuning(EPLM-HT)system is trained on an annotated conversation dataset.The proposed approach outperformed compared to the base BERT,GPT,DistilBERT and XLNet transformer models.The proposed ensemble model with the fine-tuned parameters achieved an F1_score of 0.88.

Enhancing Arabic Cyberbullying Detection with End-to-End Transformer Model

Cyberbullying,a critical concern for digital safety,necessitates effective linguistic analysis tools that can navigate the complexities of language use in online spaces.To tackle this challenge,our study introduces a new approach employing Bidirectional Encoder Representations from the Transformers(BERT)base model(cased),originally pretrained in English.This model is uniquely adapted to recognize the intricate nuances of Arabic online communication,a key aspect often overlooked in conventional cyberbullying detection methods.Our model is an end-to-end solution that has been fine-tuned on a diverse dataset of Arabic social media(SM)tweets showing a notable increase in detection accuracy and sensitivity compared to existing methods.Experimental results on a diverse Arabic dataset collected from the‘X platform’demonstrate a notable increase in detection accuracy and sensitivity compared to existing methods.E-BERT shows a substantial improvement in performance,evidenced by an accuracy of 98.45%,precision of 99.17%,recall of 99.10%,and an F1 score of 99.14%.The proposed E-BERT not only addresses a critical gap in cyberbullying detection in Arabic online forums but also sets a precedent for applying cross-lingual pretrained models in regional language applications,offering a scalable and effective framework for enhancing online safety across Arabic-speaking communities.

基于BERT-TENER的服装质量抽检通告命名实体识别

识别服装质量抽检通告中的实体信息,对于评估不同区域的服装质量状况以及制定宏观政策具有重要意义。针对质量抽检通告命名实体识别存在的长文本序列信息丢失、小类样本特征学习不全等问题,以注意力机制为核心,提出了基于BERT(bidirectional encoder representations from transformers)和TENER(transformer encoder for NER)模型的领域命名实体识别模型。BERT-TENER模型通过预训练模型BERT获得字符的动态字向量;将字向量输入TENER模块中,基于注意力机制使得同样的字符拥有不同的学习过程,基于改进的Transformer模型进一步捕捉字符与字符之间的距离和方向信息,增强模型对不同长度、小类别文本内容的理解,并采用条件随机场模型获得每个字符对应的实体标签。在领域数据集上,BERT-TENER模型针对服装抽检领域的实体识别F_1达到92.45%,相较传统方法有效提升了命名实体识别率,并且在长文本以及非均衡的实体类别中也表现出较好的性能。

基于融合策略的突发公共卫生事件网络舆情多模态负面情感识别

突发公共卫生事件以社交媒体为阵地进行线下舆情的线上映射,而图文并茂的多模态信息成为公众情感表达的主要方式。为充分利用不同模态间的关联性和互补性,提升突发公共卫生事件网络舆情多模态负面情感识别精准度,本文构建了两阶段混合融合策略驱动的多模态细粒度负面情感识别模型(two-stage,hybrid fusion strategy-driven multimodal fine-grained negative sentiment recognition model,THFMFNSR)。该模型包括多模态特征表示、特征融合、分类器和决策融合4个部分。本文通过收集新浪微博新冠肺炎的相关图文数据,验证了该模型的有效性,并抽取了最佳情感决策融合规则和分类器配置。研究结果表明,相比于文本、图像、图文特征融合的最优识别模型,本文模型在情感识别方面精确率分别提高了14.48%、12.92%、2.24%;在细粒度负面情感识别方面,精确率分别提高了22.73%、10.85%、3.34%。通过该多模态细粒度负面情感识别模型可感知舆情态势,从而辅助公共卫生部门和舆情管控部门决策。

基于图卷积神经网络的古汉语分词研究

古汉语的语法有省略、语序倒置的特点,词法有词类活用、代词名词丰富的特点,这些特点增加了古汉语分词的难度,并带来严重的out-of-vocabulary(OOV)问题。目前,深度学习方法已被广泛地应用在古汉语分词任务中并取得了成功,但是这些研究更关注的是如何提高分词效果,忽视了分词任务中的一大挑战,即OOV问题。因此,本文提出了一种基于图卷积神经网络的古汉语分词框架,通过结合预训练语言模型和图卷积神经网络,将外部知识融合到神经网络模型中来提高分词性能并缓解OOV问题。在《左传》《战国策》和《儒林外史》3个古汉语分词数据集上的研究结果显示,本文模型提高了3个数据集的分词表现。进一步的研究分析证明,本文模型能够有效地融合词典和N-gram信息;特别是N-gram有助于缓解OOV问题。

基于BERT-BiGRU模型的文本分类研究

文本分类是自然语言处理的典型应用,目前文本分类最常用的是深度学习的分类方法。针对中文文本数据具有多种特性,例如隐喻表达、语义多义性、语法特异性等,在文本分类中进行研究。提出基于编码器-解码器的双向编码表示法-双向门控制循环单元(bidirectional encoder representations from transformers-bidirectional gate recurrent unit,BERT-BiGRU)模型结构,使用BERT模型代替传统的Word2vec模型表示词向量,根据上下文信息计算字的表示,在融合上下文信息的同时还能根据字的多义性进行调整,增强了字的语义表示。在BERT模型后面增加了BiGRU,将训练后的词向量作为Bi GRU的输入进行训练,该模型可以同时从两个方向对文本信息进行特征提取,使模型具有更好的文本表示信息能力,达到更精确的文本分类效果。使用提出的BERT-BiGRU模型进行文本分类,最终准确率达到0.93,召回率达到0.94,综合评价数值F1达到0.93。通过与其他模型的试验结果对比,发现BERT-BiGRU模型在中文文本分类任务中有良好的性能。

Effective short text classification via the fusion of hybrid features for IoT social data

Nowadays short texts can be widely found in various social data in relation to the 5G-enabled Internet of Things (IoT). Short text classification is a challenging task due to its sparsity and the lack of context. Previous studies mainly tackle these problems by enhancing the semantic information or the statistical information individually. However, the improvement achieved by a single type of information is limited, while fusing various information may help to improve the classification accuracy more effectively. To fuse various information for short text classification, this article proposes a feature fusion method that integrates the statistical feature and the comprehensive semantic feature together by using the weighting mechanism and deep learning models. In the proposed method, we apply Bidirectional Encoder Representations from Transformers (BERT) to generate word vectors on the sentence level automatically, and then obtain the statistical feature, the local semantic feature and the overall semantic feature using Term Frequency-Inverse Document Frequency (TF-IDF) weighting approach, Convolutional Neural Network (CNN) and Bidirectional Gate Recurrent Unit (BiGRU). Then, the fusion feature is accordingly obtained for classification. Experiments are conducted on five popular short text classification datasets and a 5G-enabled IoT social dataset and the results show that our proposed method effectively improves the classification performance.

Deep-BERT:Transfer Learning for Classifying Multilingual Offensive Texts on Social Media

Offensive messages on social media,have recently been frequently used to harass and criticize people.In recent studies,many promising algorithms have been developed to identify offensive texts.Most algorithms analyze text in a unidirectional manner,where a bidirectional method can maximize performance results and capture semantic and contextual information in sentences.In addition,there are many separate models for identifying offensive texts based on monolin-gual and multilingual,but there are a few models that can detect both monolingual and multilingual-based offensive texts.In this study,a detection system has been developed for both monolingual and multilingual offensive texts by combining deep convolutional neural network and bidirectional encoder representations from transformers(Deep-BERT)to identify offensive posts on social media that are used to harass others.This paper explores a variety of ways to deal with multilin-gualism,including collaborative multilingual and translation-based approaches.Then,the Deep-BERT is tested on the Bengali and English datasets,including the different bidirectional encoder representations from transformers(BERT)pre-trained word-embedding techniques,and found that the proposed Deep-BERT’s efficacy outperformed all existing offensive text classification algorithms reaching an accuracy of 91.83%.The proposed model is a state-of-the-art model that can classify both monolingual-based and multilingual-based offensive texts.

A Novel Named Entity Recognition Scheme for Steel E-Commerce Platforms Using a Lite BERT

In the era of big data,E-commerce plays an increasingly important role,and steel E-commerce certainly occupies a positive position.However,it is very difficult to choose satisfactory steel raw materials from diverse steel commodities online on steel E-commerce platforms in the purchase of staffs.In order to improve the efficiency of purchasers searching for commodities on the steel E-commerce platforms,we propose a novel deep learning-based loss function for named entity recognition(NER).Considering the impacts of small sample and imbalanced data,in our NER scheme,the focal loss,the label smoothing,and the cross entropy are incorporated into a lite bidirectional encoder representations from transformers(BERT)model to avoid the over-fitting.Moreover,through the analysis of different classic annotation techniques used to tag data,an ideal one is chosen for the training model in our proposed scheme.Experiments are conducted on Chinese steel E-commerce datasets.The experimental results show that the training time of a lite BERT(ALBERT)-based method is much shorter than that of BERT-based models,while achieving the similar computational performance in terms of metrics precision,recall,and F1 with BERT-based models.Meanwhile,our proposed approach performs much better than that of combining Word2Vec,bidirectional long short-term memory(Bi-LSTM),and conditional random field(CRF)models,in consideration of training time and F1.

Text Augmentation-Based Model for Emotion Recognition Using Transformers

Emotion Recognition in Conversations(ERC)is fundamental in creating emotionally intelligentmachines.Graph-BasedNetwork(GBN)models have gained popularity in detecting conversational contexts for ERC tasks.However,their limited ability to collect and acquire contextual information hinders their effectiveness.We propose a Text Augmentation-based computational model for recognizing emotions using transformers(TA-MERT)to address this.The proposed model uses the Multimodal Emotion Lines Dataset(MELD),which ensures a balanced representation for recognizing human emotions.Themodel used text augmentation techniques to producemore training data,improving the proposed model’s accuracy.Transformer encoders train the deep neural network(DNN)model,especially Bidirectional Encoder(BE)representations that capture both forward and backward contextual information.This integration improves the accuracy and robustness of the proposed model.Furthermore,we present a method for balancing the training dataset by creating enhanced samples from the original dataset.By balancing the dataset across all emotion categories,we can lessen the adverse effects of data imbalance on the accuracy of the proposed model.Experimental results on the MELD dataset show that TA-MERT outperforms earlier methods,achieving a weighted F1 score of 62.60%and an accuracy of 64.36%.Overall,the proposed TA-MERT model solves the GBN models’weaknesses in obtaining contextual data for ERC.TA-MERT model recognizes human emotions more accurately by employing text augmentation and transformer-based encoding.The balanced dataset and the additional training samples also enhance its resilience.These findings highlight the significance of transformer-based approaches for special emotion recognition in conversations.

A Use Case of Patent Classification Using Deep Learning with Transfer Learning

Purpose:Patent classification is one of the areas in Intellectual Property Analytics(IPA),and a growing use case since the number of patent applications has been increasing worldwide.We propose using machine learning algorithms to classify Portuguese patents and evaluate the performance of transfer learning methodologies to solve this task.Design/methodology/approach:We applied three different approaches in this paper.First,we used a dataset available by INPI to explore traditional machine learning algorithms and ensemble methods.After preprocessing data by applying TF-IDF,FastText and Doc2Vec,the models were evaluated by cross-validation in 5 folds.In a second approach,we used two different Neural Networks architectures,a Convolutional Neural Network(CNN)and a bi-directional Long Short-Term Memory(BiLSTM).Finally,we used pre-trained BERT,DistilBERT,and ULMFiT models in the third approach.Findings:BERTTimbau,a BERT architecture model pre-trained on a large Portuguese corpus,presented the best results for the task,even though with a performance of only 4%superior to a LinearSVC model using TF-IDF feature engineering.Research limitations:The dataset was highly imbalanced,as usual in patent applications,so the classes with the lowest samples were expected to present the worst performance.That result happened in some cases,especially in classes with less than 60 training samples.Practical implications:Patent classification is challenging because of the hierarchical classification system,the context overlap,and the underrepresentation of the classes.However,the final model presented an acceptable performance given the size of the dataset and the task complexity.This model can support the decision and improve the time by proposing a category in the second level of ICP,which is one of the critical phases of the grant patent process.Originality/value:To our knowledge,the proposed models were never implemented for Portuguese patent classification.

End-to-end aspect category sentiment analysis based on type graph convolutional networks

For the existing aspect category sentiment analysis research,most of the aspects are given for sentiment extraction,and this pipeline method is prone to error accumulation,and the use of graph convolutional neural network for aspect category sentiment analysis does not fully utilize the dependency type information between words,so it cannot enhance feature extraction.This paper proposes an end-to-end aspect category sentiment analysis(ETESA)model based on type graph convolutional networks.The model uses the bidirectional encoder representation from transformers(BERT)pretraining model to obtain aspect categories and word vectors containing contextual dynamic semantic information,which can solve the problem of polysemy;when using graph convolutional network(GCN)for feature extraction,the fusion operation of word vectors and initialization tensor of dependency types can obtain the importance values of different dependency types and enhance the text feature representation;by transforming aspect category and sentiment pair extraction into multiple single-label classification problems,aspect category and sentiment can be extracted simultaneously in an end-to-end way and solve the problem of error accumulation.Experiments are tested on three public datasets,and the results show that the ETESA model can achieve higher Precision,Recall and F1 value,proving the effectiveness of the model.

AN IMPROVED ALGORITHM OF GMM VOICE CONVERSION SYSTEM BASED ON CHANGING THE TIME-SCALE

This paper improves and presents an advanced method of the voice conversion system based on Gaussian Mixture Models(GMM) models by changing the time-scale of speech.The Speech Transformation and Representation using Adaptive Interpolation of weiGHTed spectrum(STRAIGHT) model is adopted to extract the spectrum features,and the GMM models are trained to generate the conversion function.The spectrum features of a source speech will be converted by the conversion function.The time-scale of speech is changed by extracting the converted features and adding to the spectrum.The conversion voice was evaluated by subjective and objective measurements.The results confirm that the transformed speech not only approximates the characteristics of the target speaker,but also more natural and more intelligible.

Comparative Analysis of Machine Learning Algorithms for Email Phishing Detection Using TF-IDF, Word2Vec, and BERT

Cybercriminals often use fraudulent emails and fictitious email accounts to deceive individuals into disclosing confidential information,a practice known as phishing.This study utilizes three distinct methodologies,Term Frequency-Inverse Document Frequency,Word2Vec,and Bidirectional Encoder Representations from Transform-ers,to evaluate the effectiveness of various machine learning algorithms in detecting phishing attacks.The study uses feature extraction methods to assess the performance of Logistic Regression,Decision Tree,Random Forest,and Multilayer Perceptron algorithms.The best results for each classifier using Term Frequency-Inverse Document Frequency were Multilayer Perceptron(Precision:0.98,Recall:0.98,F1-score:0.98,Accuracy:0.98).Word2Vec’s best results were Multilayer Perceptron(Precision:0.98,Recall:0.98,F1-score:0.98,Accuracy:0.98).The highest performance was achieved using the Bidirectional Encoder Representations from the Transformers model,with Precision,Recall,F1-score,and Accuracy all reaching 0.99.This study highlights how advanced pre-trained models,such as Bidirectional Encoder Representations from Transformers,can significantly enhance the accuracy and reliability of fraud detection systems.

Traditional Chinese Medicine Synonymous Term Conversion:A Bidirectional Encoder Representations from Transformers-Based Model for Converting Synonymous Terms in Traditional Chinese Medicine

Background:The medical records of traditional Chinese medicine(TCM)contain numerous synonymous terms with different descriptions,which is not conducive to computer-aided data mining of TCM.However,there is a lack of models available to normalize synonymous TCM terms.Therefore,construction of a synonymous term conversion(STC)model for normalizing synonymous TCM terms is necessary.Methods:Based on the neural networks of bidirectional encoder representations from transformers(BERT),four types of TCM STC models were designed:Models based on BERT and text classification,text sequence generation,named entity recognition,and text matching.The superior STC model was selected on the basis of its performance in converting synonymous terms.Moreover,three misjudgment inspection methods for the conversion results of the STC model based on inconsistency were proposed to find incorrect term conversion:Neuron random deactivation,output comparison of multiple isomorphic models,and output comparison of multiple heterogeneous models(OCMH).Results:The classification-based STC model outperformed the other STC task models.It achieved F1 scores of 0.91,0.91,and 0.83 for performing symptoms,patterns,and treatments STC tasks,respectively.The OCMH method showed the best performance in misjudgment inspection,with wrong detection rates of 0.80,0.84,and 0.90 in the term conversion results for symptoms,patterns,and treatments,respectively.Conclusion:The TCM STC model based on classification achieved superior performance in converting synonymous terms for symptoms,patterns,and treatments.The misjudgment inspection method based on OCMH showed superior performance in identifying incorrect outputs.

基于BERT的阅读理解式标书文本信息抽取方法

针对标书文本重要信息的抽取需求,提出一种基于BERT(bidirectional encoder representations from transformers)的阅读理解式标书文本信息抽取方法。该方法将信息抽取任务转换为阅读理解任务,根据标书文本内容,生成对应问题,再抽取标书文本片段作为问题答案。利用BERT预训练模型,得到强健的语言模型,获取更深层次的上下文关联。相比传统的命名实体识别方法,基于阅读理解的信息抽取方法能够很好地同时处理非嵌套实体和嵌套实体的抽取,也能充分利用问题所包含的先验语义信息,区分出具有相似属性的信息。从中国政府采购网下载标书文本数据进行了实验,本文方法总体EM(exact match)值达到92.41%,F1值达到95.03%。实验结果表明本文提出的方法对标书文本的信息抽取是有效的。