Traditional Chinese Medicine Synonymous Term Conversion:A Bidirectional Encoder Representations from Transformers-Based Model for Converting Synonymous Terms in Traditional Chinese Medicine

Background:The medical records of traditional Chinese medicine(TCM)contain numerous synonymous terms with different descriptions,which is not conducive to computer-aided data mining of TCM.However,there is a lack of models available to normalize synonymous TCM terms.Therefore,construction of a synonymous term conversion(STC)model for normalizing synonymous TCM terms is necessary.Methods:Based on the neural networks of bidirectional encoder representations from transformers(BERT),four types of TCM STC models were designed:Models based on BERT and text classification,text sequence generation,named entity recognition,and text matching.The superior STC model was selected on the basis of its performance in converting synonymous terms.Moreover,three misjudgment inspection methods for the conversion results of the STC model based on inconsistency were proposed to find incorrect term conversion:Neuron random deactivation,output comparison of multiple isomorphic models,and output comparison of multiple heterogeneous models(OCMH).Results:The classification-based STC model outperformed the other STC task models.It achieved F1 scores of 0.91,0.91,and 0.83 for performing symptoms,patterns,and treatments STC tasks,respectively.The OCMH method showed the best performance in misjudgment inspection,with wrong detection rates of 0.80,0.84,and 0.90 in the term conversion results for symptoms,patterns,and treatments,respectively.Conclusion:The TCM STC model based on classification achieved superior performance in converting synonymous terms for symptoms,patterns,and treatments.The misjudgment inspection method based on OCMH showed superior performance in identifying incorrect outputs.

基于BERT与细粒度特征提取的数据法学问答系统

首先利用bidirectional encoder representations from transformers(BERT)模型的强大的语境理解能力来提取数据法律文本的深层语义特征,然后引入细粒度特征提取层,依照注意力机制,重点关注文本中与数据法律问答相关的关键部分,最后对所采集的法律问答数据集进行训练和评估.结果显示:与传统的多个单一模型相比,所提出的模型在准确度、精确度、召回率、F1分数等关键性能指标上均有提升,表明该系统能够更有效地理解和回应复杂的数据法学问题,为研究数据法学的专业人士和公众用户提供更高质量的问答服务.

Enhanced Topic-Aware Summarization Using Statistical Graph Neural Networks

The rapid expansion of online content and big data has precipitated an urgent need for efficient summarization techniques to swiftly comprehend vast textual documents without compromising their original integrity.Current approaches in Extractive Text Summarization(ETS)leverage the modeling of inter-sentence relationships,a task of paramount importance in producing coherent summaries.This study introduces an innovative model that integrates Graph Attention Networks(GATs)with Transformer-based Bidirectional Encoder Representa-tions from Transformers(BERT)and Latent Dirichlet Allocation(LDA),further enhanced by Term Frequency-Inverse Document Frequency(TF-IDF)values,to improve sentence selection by capturing comprehensive topical information.Our approach constructs a graph with nodes representing sentences,words,and topics,thereby elevating the interconnectivity and enabling a more refined understanding of text structures.This model is stretched to Multi-Document Summarization(MDS)from Single-Document Summarization,offering significant improvements over existing models such as THGS-GMM and Topic-GraphSum,as demonstrated by empirical evaluations on benchmark news datasets like Cable News Network(CNN)/Daily Mail(DM)and Multi-News.The results consistently demonstrate superior performance,showcasing the model’s robustness in handling complex summarization tasks across single and multi-document contexts.This research not only advances the integration of BERT and LDA within a GATs but also emphasizes our model’s capacity to effectively manage global information and adapt to diverse summarization challenges.

BSTFNet:An Encrypted Malicious Traffic Classification Method Integrating Global Semantic and Spatiotemporal Features

While encryption technology safeguards the security of network communications,malicious traffic also uses encryption protocols to obscure its malicious behavior.To address the issues of traditional machine learning methods relying on expert experience and the insufficient representation capabilities of existing deep learning methods for encrypted malicious traffic,we propose an encrypted malicious traffic classification method that integrates global semantic features with local spatiotemporal features,called BERT-based Spatio-Temporal Features Network(BSTFNet).At the packet-level granularity,the model captures the global semantic features of packets through the attention mechanism of the Bidirectional Encoder Representations from Transformers(BERT)model.At the byte-level granularity,we initially employ the Bidirectional Gated Recurrent Unit(BiGRU)model to extract temporal features from bytes,followed by the utilization of the Text Convolutional Neural Network(TextCNN)model with multi-sized convolution kernels to extract local multi-receptive field spatial features.The fusion of features from both granularities serves as the ultimate multidimensional representation of malicious traffic.Our approach achieves accuracy and F1-score of 99.39%and 99.40%,respectively,on the publicly available USTC-TFC2016 dataset,and effectively reduces sample confusion within the Neris and Virut categories.The experimental results demonstrate that our method has outstanding representation and classification capabilities for encrypted malicious traffic.

基于融合策略的突发公共卫生事件网络舆情多模态负面情感识别

突发公共卫生事件以社交媒体为阵地进行线下舆情的线上映射,而图文并茂的多模态信息成为公众情感表达的主要方式。为充分利用不同模态间的关联性和互补性,提升突发公共卫生事件网络舆情多模态负面情感识别精准度,本文构建了两阶段混合融合策略驱动的多模态细粒度负面情感识别模型(two-stage,hybrid fusion strategy-driven multimodal fine-grained negative sentiment recognition model,THFMFNSR)。该模型包括多模态特征表示、特征融合、分类器和决策融合4个部分。本文通过收集新浪微博新冠肺炎的相关图文数据,验证了该模型的有效性,并抽取了最佳情感决策融合规则和分类器配置。研究结果表明,相比于文本、图像、图文特征融合的最优识别模型,本文模型在情感识别方面精确率分别提高了14.48%、12.92%、2.24%;在细粒度负面情感识别方面,精确率分别提高了22.73%、10.85%、3.34%。通过该多模态细粒度负面情感识别模型可感知舆情态势,从而辅助公共卫生部门和舆情管控部门决策。

基于图卷积神经网络的古汉语分词研究

古汉语的语法有省略、语序倒置的特点,词法有词类活用、代词名词丰富的特点,这些特点增加了古汉语分词的难度,并带来严重的out-of-vocabulary(OOV)问题。目前,深度学习方法已被广泛地应用在古汉语分词任务中并取得了成功,但是这些研究更关注的是如何提高分词效果,忽视了分词任务中的一大挑战,即OOV问题。因此,本文提出了一种基于图卷积神经网络的古汉语分词框架,通过结合预训练语言模型和图卷积神经网络,将外部知识融合到神经网络模型中来提高分词性能并缓解OOV问题。在《左传》《战国策》和《儒林外史》3个古汉语分词数据集上的研究结果显示,本文模型提高了3个数据集的分词表现。进一步的研究分析证明,本文模型能够有效地融合词典和N-gram信息;特别是N-gram有助于缓解OOV问题。

Deep-BERT:Transfer Learning for Classifying Multilingual Offensive Texts on Social Media

Offensive messages on social media,have recently been frequently used to harass and criticize people.In recent studies,many promising algorithms have been developed to identify offensive texts.Most algorithms analyze text in a unidirectional manner,where a bidirectional method can maximize performance results and capture semantic and contextual information in sentences.In addition,there are many separate models for identifying offensive texts based on monolin-gual and multilingual,but there are a few models that can detect both monolingual and multilingual-based offensive texts.In this study,a detection system has been developed for both monolingual and multilingual offensive texts by combining deep convolutional neural network and bidirectional encoder representations from transformers(Deep-BERT)to identify offensive posts on social media that are used to harass others.This paper explores a variety of ways to deal with multilin-gualism,including collaborative multilingual and translation-based approaches.Then,the Deep-BERT is tested on the Bengali and English datasets,including the different bidirectional encoder representations from transformers(BERT)pre-trained word-embedding techniques,and found that the proposed Deep-BERT’s efficacy outperformed all existing offensive text classification algorithms reaching an accuracy of 91.83%.The proposed model is a state-of-the-art model that can classify both monolingual-based and multilingual-based offensive texts.

End-to-end aspect category sentiment analysis based on type graph convolutional networks

For the existing aspect category sentiment analysis research,most of the aspects are given for sentiment extraction,and this pipeline method is prone to error accumulation,and the use of graph convolutional neural network for aspect category sentiment analysis does not fully utilize the dependency type information between words,so it cannot enhance feature extraction.This paper proposes an end-to-end aspect category sentiment analysis(ETESA)model based on type graph convolutional networks.The model uses the bidirectional encoder representation from transformers(BERT)pretraining model to obtain aspect categories and word vectors containing contextual dynamic semantic information,which can solve the problem of polysemy;when using graph convolutional network(GCN)for feature extraction,the fusion operation of word vectors and initialization tensor of dependency types can obtain the importance values of different dependency types and enhance the text feature representation;by transforming aspect category and sentiment pair extraction into multiple single-label classification problems,aspect category and sentiment can be extracted simultaneously in an end-to-end way and solve the problem of error accumulation.Experiments are tested on three public datasets,and the results show that the ETESA model can achieve higher Precision,Recall and F1 value,proving the effectiveness of the model.

基于BERT-BiGRU模型的文本分类研究

文本分类是自然语言处理的典型应用,目前文本分类最常用的是深度学习的分类方法。针对中文文本数据具有多种特性,例如隐喻表达、语义多义性、语法特异性等,在文本分类中进行研究。提出基于编码器-解码器的双向编码表示法-双向门控制循环单元(bidirectional encoder representations from transformers-bidirectional gate recurrent unit,BERT-BiGRU)模型结构,使用BERT模型代替传统的Word2vec模型表示词向量,根据上下文信息计算字的表示,在融合上下文信息的同时还能根据字的多义性进行调整,增强了字的语义表示。在BERT模型后面增加了BiGRU,将训练后的词向量作为Bi GRU的输入进行训练,该模型可以同时从两个方向对文本信息进行特征提取,使模型具有更好的文本表示信息能力,达到更精确的文本分类效果。使用提出的BERT-BiGRU模型进行文本分类,最终准确率达到0.93,召回率达到0.94,综合评价数值F1达到0.93。通过与其他模型的试验结果对比,发现BERT-BiGRU模型在中文文本分类任务中有良好的性能。

A Novel Named Entity Recognition Scheme for Steel E-Commerce Platforms Using a Lite BERT

In the era of big data,E-commerce plays an increasingly important role,and steel E-commerce certainly occupies a positive position.However,it is very difficult to choose satisfactory steel raw materials from diverse steel commodities online on steel E-commerce platforms in the purchase of staffs.In order to improve the efficiency of purchasers searching for commodities on the steel E-commerce platforms,we propose a novel deep learning-based loss function for named entity recognition(NER).Considering the impacts of small sample and imbalanced data,in our NER scheme,the focal loss,the label smoothing,and the cross entropy are incorporated into a lite bidirectional encoder representations from transformers(BERT)model to avoid the over-fitting.Moreover,through the analysis of different classic annotation techniques used to tag data,an ideal one is chosen for the training model in our proposed scheme.Experiments are conducted on Chinese steel E-commerce datasets.The experimental results show that the training time of a lite BERT(ALBERT)-based method is much shorter than that of BERT-based models,while achieving the similar computational performance in terms of metrics precision,recall,and F1 with BERT-based models.Meanwhile,our proposed approach performs much better than that of combining Word2Vec,bidirectional long short-term memory(Bi-LSTM),and conditional random field(CRF)models,in consideration of training time and F1.

基于BERT的阅读理解式标书文本信息抽取方法

针对标书文本重要信息的抽取需求,提出一种基于BERT(bidirectional encoder representations from transformers)的阅读理解式标书文本信息抽取方法。该方法将信息抽取任务转换为阅读理解任务,根据标书文本内容,生成对应问题,再抽取标书文本片段作为问题答案。利用BERT预训练模型,得到强健的语言模型,获取更深层次的上下文关联。相比传统的命名实体识别方法,基于阅读理解的信息抽取方法能够很好地同时处理非嵌套实体和嵌套实体的抽取,也能充分利用问题所包含的先验语义信息,区分出具有相似属性的信息。从中国政府采购网下载标书文本数据进行了实验,本文方法总体EM(exact match)值达到92.41%,F1值达到95.03%。实验结果表明本文提出的方法对标书文本的信息抽取是有效的。

Intelligent Prescription-Generating Models of Traditional Chinese Medicine Based on Deep Learning

Objective:This study aimed to construct an intelligent prescription-generating(IPG)model based on deep-learning natural language processing(NLP)technology for multiple prescriptions in Chinese medicine.Materials and Methods:We selected the Treatise on Febrile Diseases and the Synopsis of Golden Chamber as basic datasets with EDA data augmentation,and the Yellow Emperor’s Canon of Internal Medicine,the Classic of the Miraculous Pivot,and the Classic on Medical Problems as supplementary datasets for fine-tuning.We selected the word-embedding model based on the Imperial Collection of Four,the bidirectional encoder representations from transformers(BERT)model based on the Chinese Wikipedia,and the robustly optimized BERT approach(RoBERTa)model based on the Chinese Wikipedia and a general database.In addition,the BERT model was fine-tuned using the supplementary datasets to generate a Traditional Chinese Medicine-BERT model.Multiple IPG models were constructed based on the pretraining strategy and experiments were performed.Metrics of precision,recall,and F1-score were used to assess the model performance.Based on the trained models,we extracted and visualized the semantic features of some typical texts from treatise on febrile diseases and investigated the patterns.Results:Among all the trained models,the RoBERTa-large model performed the best,with a test set precision of 92.22%,recall of 86.71%,and F1-score of 89.38%and 10-fold cross-validation precision of 94.5%±2.5%,recall of 90.47%±4.1%,and F1-score of 92.38%±2.8%.The semantic feature extraction results based on this model showed that the model was intelligently stratified based on different meanings such that the within-layer’s patterns showed the associations of symptom–symptoms,disease–symptoms,and symptom–punctuations,while the between-layer’s patterns showed a progressive or dynamic symptom and disease transformation.Conclusions:Deep-learning-based NLP technology significantly improves the performance of IPG model.In addition,NLP-based semantic feature extraction may be vital to further investigate the ancient Chinese medicine texts.

Research on sentiment terms extraction and visualization of character sentimental interactions in A Dream of Red Mansions

In the context of interdisciplinary research,using computer technology to further mine keywords in cultural texts and carry out semantic analysis can deepen the understanding of texts,and provide quantitative support and evidence for humanistic studies.Based on the novel A Dream of Red Mansions,the automatic extraction and classification of those sentiment terms in it were realized,and detailed analysis of large-scale sentiment terms was carried out.Bidirectional encoder representation from transformers(BERT) pretraining and fine-tuning model was used to construct the sentiment classifier of A Dream of Red Mansions.Sentiment terms of A Dream of Red Mansions are divided into eight sentimental categories,and the relevant people in sentences are extracted according to specific rules.It also tries to visually display the sentimental interactions between Twelve Girls of Jinling and Jia Baoyu along with the development of the episode.The overall F_(1) score of BERT-based sentiment classifier reached 84.89%.The best single sentiment score reached 91.15%.Experimental results show that the classifier can satisfactorily classify the text of A Dream of Red Mansions,and the text classification and interactional analysis results can be mutually verified with the text interpretation of A dream of Red Mansions by literature experts.