PAL-BERT:An Improved Question Answering Model

In the field of natural language processing(NLP),there have been various pre-training language models in recent years,with question answering systems gaining significant attention.However,as algorithms,data,and computing power advance,the issue of increasingly larger models and a growing number of parameters has surfaced.Consequently,model training has become more costly and less efficient.To enhance the efficiency and accuracy of the training process while reducing themodel volume,this paper proposes a first-order pruningmodel PAL-BERT based on the ALBERT model according to the characteristics of question-answering(QA)system and language model.Firstly,a first-order network pruning method based on the ALBERT model is designed,and the PAL-BERT model is formed.Then,the parameter optimization strategy of the PAL-BERT model is formulated,and the Mish function was used as an activation function instead of ReLU to improve the performance.Finally,after comparison experiments with traditional deep learning models TextCNN and BiLSTM,it is confirmed that PALBERT is a pruning model compression method that can significantly reduce training time and optimize training efficiency.Compared with traditional models,PAL-BERT significantly improves the NLP task’s performance.

Operational requirements analysis method based on question answering of WEKG

The weapon and equipment operational requirement analysis(WEORA) is a necessary condition to win a future war,among which the acquisition of knowledge about weapons and equipment is a great challenge. The main challenge is that the existing weapons and equipment data fails to carry out structured knowledge representation, and knowledge navigation based on natural language cannot efficiently support the WEORA. To solve above problem, this research proposes a method based on question answering(QA) of weapons and equipment knowledge graph(WEKG) to construct and navigate the knowledge related to weapons and equipment in the WEORA. This method firstly constructs the WEKG, and builds a neutral network-based QA system over the WEKG by means of semantic parsing for knowledge navigation. Finally, the method is evaluated and a chatbot on the QA system is developed for the WEORA. Our proposed method has good performance in the accuracy and efficiency of searching target knowledge, and can well assist the WEORA.

Analysis of community question-answering issues via machine learning and deep learning:State-of-the-art review

Over the last couple of decades,community question-answering sites(CQAs)have been a topic of much academic interest.Scholars have often leveraged traditional machine learning(ML)and deep learning(DL)to explore the ever-growing volume of content that CQAs engender.To clarify the current state of the CQA literature that has used ML and DL,this paper reports a systematic literature review.The goal is to summarise and synthesise the major themes of CQA research related to(i)questions,(ii)answers and(iii)users.The final review included 133 articles.Dominant research themes include question quality,answer quality,and expert identification.In terms of dataset,some of the most widely studied platforms include Yahoo!Answers,Stack Exchange and Stack Overflow.The scope of most articles was confined to just one platform with few cross-platform investigations.Articles with ML outnumber those with DL.Nonetheless,the use of DL in CQA research is on an upward trajectory.A number of research directions are proposed.

Deep Multi-Module Based Language Priors Mitigation Model for Visual Question Answering

The original intention of visual question answering(VQA)models is to infer the answer based on the relevant information of the question text in the visual image,but many VQA models often yield answers that are biased by some prior knowledge,especially the language priors.This paper proposes a mitigation model called language priors mitigation-VQA(LPM-VQA)for the language priors problem in VQA model,which divides language priors into positive and negative language priors.Different network branches are used to capture and process the different priors to achieve the purpose of mitigating language priors.A dynamically-changing language prior feedback objective function is designed with the intermediate results of some modules in the VQA model.The weight of the loss value for each answer is dynamically set according to the strength of its language priors to balance its proportion in the total VQA loss to further mitigate the language priors.This model does not depend on the baseline VQA architectures and can be configured like a plug-in to improve the performance of the model over most existing VQA models.The experimental results show that the proposed model is general and effective,achieving state-of-the-art accuracy in the VQA-CP v2 dataset.

ALBERT with Knowledge Graph Encoder Utilizing Semantic Similarity for Commonsense Question Answering

Recently,pre-trained language representation models such as bidirec-tional encoder representations from transformers(BERT)have been performing well in commonsense question answering(CSQA).However,there is a problem that the models do not directly use explicit information of knowledge sources existing outside.To augment this,additional methods such as knowledge-aware graph network(KagNet)and multi-hop graph relation network(MHGRN)have been proposed.In this study,we propose to use the latest pre-trained language model a lite bidirectional encoder representations from transformers(ALBERT)with knowledge graph information extraction technique.We also propose to applying the novel method,schema graph expansion to recent language models.Then,we analyze the effect of applying knowledge graph-based knowledge extraction techniques to recent pre-trained language models and confirm that schema graph expansion is effective in some extent.Furthermore,we show that our proposed model can achieve better performance than existing KagNet and MHGRN models in CommonsenseQA dataset.

PDID:视觉离散化智能问答模型——基于图像像素离散化和图像语义离散化的VQA模型

视觉问答是一项具有挑战性的多模态任务,它连接了计算机视觉和自然语言处理两个领域。在这项任务中,模型需要根据给定的图片和相关问题,有效地提取信息并给出正确答案。然而,由于图像和文本属于不同的模态,存在着严重的语义差异,因此如何有效地将不同模态的信息对齐并减少语义差异,是当前视觉问答领域的重点关注问题。本文针对当前视觉问答方法在多模态对齐阶段图像和文本信息颗粒度的巨大差异,提出了基于视觉离散化(PDID: Pixel Discretization and Instance Discretization)的智能问答模型并辅助以模态注意力机制完成跨模态信息和语义对齐。图像以像素为最小单位的特征数据与文本以单词为最小单位的特征数据,它们在数据的信息颗粒度上存在巨大的差异,即语言通过至多数万单词即可完成整个文本语义空间的构建,而图像则是通过亿级的RGB三原色数组构建而成。这说明了直接建模以像素为单位的图像是很难和文本做好对齐的。本文通过了多种图像离散化的方式,一方面通过离散化图像像素,以颜色离散化、强度离散化、纹理离散化、空间离散化四种形式将图像像素完成离散化,在数量级上逼近文本特征的最小基元数量;另一方面通过图像语义特征的软编码,离散化图像深层次的语义特征,将图像的语义特征与文本的单词语义对齐,在语义层面上逼近文本特征的单词语义信息量。除此以外,本文提出了一种新型的视觉关系融合模块,视觉关系融合模块用来捕获同种模态内离散化特征和连续特征的交互信息,为模型提供丰富的视觉特征。本文先使用自注意力方法提取模态内特征之间的相关性,即提取视觉全局关系,再使用通道空间分离注意力进行跨模态结合,为局部引导的全局特征提供更大的表示空间和更多的补充信息。为了验证本方法的有效性,在VQA-v2,COCO-QA,VQA-CP v2数据集上进行了广泛实验,充分验证了该方法在视觉问答任务中的基于离散机制的视觉问答研究有效性。同时也体现了该模型在其他跨模态任务(图像文本匹配、指示表达)中仍有很强的泛化能力。

Information Extraction Based on Multi-turn Question Answering for Analyzing Korean Research Trends

Analyzing Research and Development(R&D)trends is important because it can influence future decisions regarding R&D direction.In typical trend analysis,topic or technology taxonomies are employed to compute the popularities of the topics or codes over time.Although it is simple and effective,the taxonomies are difficult to manage because new technologies are introduced rapidly.Therefore,recent studies exploit deep learning to extract pre-defined targets such as problems and solutions.Based on the recent advances in question answering(QA)using deep learning,we adopt a multi-turn QA model to extract problems and solutions from Korean R&D reports.With the previous research,we use the reports directly and analyze the difficulties in handling them using QA style on Information Extraction(IE)for sentence-level benchmark dataset.After investigating the characteristics of Korean R&D,we propose a model to deal with multiple and repeated appearances of targets in the reports.Accordingly,we propose a model that includes an algorithm with two novel modules and a prompt.A newly proposed methodology focuses on reformulating a question without a static template or pre-defined knowledge.We show the effectiveness of the proposed model using a Korean R&D report dataset that we constructed and presented an in-depth analysis of the benefits of the multi-turn QA model.

Expert Recommendation in Community Question Answering via Heterogeneous Content Network Embedding

ExpertRecommendation(ER)aims to identify domain experts with high expertise and willingness to provide answers to questions in Community Question Answering(CQA)web services.How to model questions and users in the heterogeneous content network is critical to this task.Most traditional methods focus on modeling questions and users based on the textual content left in the community while ignoring the structural properties of heterogeneous CQA networks and always suffering from textual data sparsity issues.Recent approaches take advantage of structural proximities between nodes and attempt to fuse the textual content of nodes for modeling.However,they often fail to distinguish the nodes’personalized preferences and only consider the textual content of a part of the nodes in network embedding learning,while ignoring the semantic relevance of nodes.In this paper,we propose a novel framework that jointly considers the structural proximity relations and textual semantic relevance to model users and questions more comprehensively.Specifically,we learn topology-based embeddings through a hierarchical attentive network learning strategy,in which the proximity information and the personalized preference of nodes are encoded and preserved.Meanwhile,we utilize the node’s textual content and the text correlation between adjacent nodes to build the content-based embedding through a meta-context-aware skip-gram model.In addition,the user’s relative answer quality is incorporated to promote the ranking performance.Experimental results show that our proposed framework consistently and significantly outperforms the state-of-the-art baselines on three real-world datasets by taking the deep semantic understanding and structural feature learning together.The performance of the proposed work is analyzed in terms of MRR,P@K,and MAP and is proven to be more advanced than the existing methodologies.

Improved Blending Attention Mechanism in Visual Question Answering

Visual question answering(VQA)has attracted more and more attention in computer vision and natural language processing.Scholars are committed to studying how to better integrate image features and text features to achieve better results in VQA tasks.Analysis of all features may cause information redundancy and heavy computational burden.Attention mechanism is a wise way to solve this problem.However,using single attention mechanism may cause incomplete concern of features.This paper improves the attention mechanism method and proposes a hybrid attention mechanism that combines the spatial attention mechanism method and the channel attention mechanism method.In the case that the attention mechanism will cause the loss of the original features,a small portion of image features were added as compensation.For the attention mechanism of text features,a selfattention mechanism was introduced,and the internal structural features of sentences were strengthened to improve the overall model.The results show that attention mechanism and feature compensation add 6.1%accuracy to multimodal low-rank bilinear pooling network.

一种面向中文自动问答的注意力交互深度学习模型

随着互联网、大数据的飞速发展,以深度神经网络(DNN)为代表的人工智能技术迎来了黄金发展时期,自动问答作为人工智能领域的一个重要分支,也得到越来越多学者的关注。现有网络模型可以提取问题或答案的语义特征,但其一方面忽略了问题与答案之间的语义联系,另一方面也不能从整体上把握问题或答案内部所有字符之间的潜在联系。基于此,提出了两种不同形式的注意力交互模块,即互注意力交互模块和自注意力交互模块,并设计出一套基于所提注意力交互模块的深度学习模型,用于证明该注意力交互模块的有效性。首先将问题和答案中的每个字符映射成固定长度的向量,分别得到问题和答案对应的字嵌入矩阵;然后将字嵌入矩阵送入注意力交互模块,得到综合考虑问题与答案所有字符之后的字嵌入矩阵,并与之前的字嵌入矩阵相加,送入深度神经网络模块,用于提取问题与答案的语义特征;最后得到问题与答案的向量表示并计算两者之间的相似度。实验结果表明,所提模型的Top-1准确度较主流深度学习模型最高提升了3.55%,证明了所提注意力交互模块对于改善上述问题的有效性。

融入三维语义特征的常识推理问答方法

现有使用预训练语言模型和知识图谱的常识问答方法主要集中于构建知识图谱子图及跨模态信息结合的研究,忽略了知识图谱自身丰富的语义特征,且缺少对不同问答任务的知识图谱子图节点相关性的动态调整,导致预测准确率低。为解决以上问题,提出一种融入三维语义特征的常识推理问答方法。首先提出知识图谱节点的关系层级、实体层级、三元组层级三维语义特征量化指标;其次,通过注意力机制动态计算关系层级、实体层级、三元组层级三种维度的语义特征对不同实体节点间的重要性;最后,通过图神经网络进行多层聚合迭代嵌入三维语义特征,获得更多的外推知识表示,更新知识图谱子图节点表示,提升答案预测精度。与QA-GNN常识问答推理方法相比,所提方法在CommonsenseQA数据集上的验证集和测试集的准确率分别提高了1.70个百分点和0.74个百分点,在OpenBookQA数据集上使用AristoRoBERTa数据处理方法的准确率提高了1.13个百分点。实验结果表明,所提出的融入三维语义特征的常识推理问答方法能够有效提高常识问答任务准确率。

旅游自动问答系统中多任务问句分类研究

目前旅游产业信息化建设需要构建旅游自动问答系统,其中问句分类是问答系统的重要组成部分,传统问句类别体系角度单一,且传统分类模型对不平衡的问句数据集表现欠佳。针对这一问题,该文从问题主题和问句答案类型两个角度构建了旅游领域的问句类别体系架构,并提出多任务问句分类模型MT-Bert,在BERT上进行多任务训练,并加入自注意力机制,使用Softmax分类器,并设计了多任务融合损失函数。在山西旅游数据集的结果表明,MT-Bert在两种类别体系的微平均F1值分别为97.6%、91.7%,且避免了非平衡数据的预测失败问题,可以有效处理非平衡数据。

结合对比学习的图像指导增强视觉问答模型

针对现有的注意力编解码视觉问答模型存在两个问题:单一形态图像特征包含视觉信息不完整,以及对问题指导过度依赖,提出结合对比学习的图像指导增强视觉问答模型。所提模型包含一种双特征视觉解码器,它基于Transformer语言编码器实现,将单一的图像特征扩展为区域和网格两种形态,根据不同形态特征的相对位置构建互补的空间关系,以解决第一问题。所提模型包含一种视觉引导的语言解码器,将视觉解码的两种图像特征与问题特征二次匹配,通过平行门控引导注意力,自适应地修正不同视觉信息对问题的引导比例,以解决第二问题。所提模型,在训练过程中,引入对比学习损失函数,通过对比模型推理时不同模态特征在隐空间内的相似度,获取更相近的互信息。所提模型,在VQA 2.0、COCO-QA和GQA数据集上分别取得73.82%、72.49%和57.44%的总体准确率,较MCAN模型分别提高2.92个百分点、4.41个百分点和0.8个百分点。大量消融实验和可视化分析证明了模型的有效性。实验结果表明,所提模型能够获取更相关的语言-视觉信息,并且对不同类型的问题样本具有更强的泛化能力。

融合GPT和知识图谱的洪涝应急决策智能问答系统研究

为提高生成式预训练语言大模型(generative pre-trained transformer, GPT)的应急管理信息分析能力,以实现洪涝灾害应急处置过程中的在线辅助决策,提出融合GPT和知识图谱的应急决策智能问答系统(KG-GPT)。改进GPT架构以识别问题中的关键信息,利用知识图谱推理应急领域知识并生成具有逻辑性的回答;结合洪涝灾害的实际应急决策问答数据集并编制演练脚本,使用自动评估和专家评估方法将本系统与GPT进行对比实验。研究结果表明:该系统成功融合应急领域知识图谱和GPT模型,能够深刻理解问题的背景信息并生成流畅回答;与GPT相比,该系统可为决策者提供更快速准确的在线辅助决策工具。研究结果可提升洪涝灾害应急信息分析和决策效率。

破解历史周期率“两个答案”的辩证思考

中国共产党在跳出治乱兴衰历史周期率问题上长期求解,先后得出了人民监督的“第一个答案”和自我革命的“第二个答案”,回应了马克思主义政党如何加强自身建设和实现长期执政的重大时代课题,是对中国共产党执政规律、自身建设规律和人类社会发展规律的科学把握。“两个答案”并非孤立存在,而是相互联系、相互影响、相互制约的有机统一体。二者虽提出有先后,但同根同源、实践同步;虽主体有差异,但旨归统一、立场一致;虽内容有侧重,但任务趋同、目标相通;虽动因分内外,但机制互补、辩证互成。以辩证思维对破解历史周期率“两个答案”之间的关系进行深入思考,坚持“人民监督”与强化“自我革命”统筹推进,既是巩固党的长期执政地位的内在要求,也是走好新的赶考之路的必然选择。

基于知识图谱的问答系统研究

随着信息时代的快速发展,问答系统作为知识获取的重要工具,其研究和应用价值日益凸显。传统的问答系统主要依赖于关键词匹配或预先定义模板,难以处理复杂、具有深度的问题,而基于知识图谱的问答系统应用先进的语言系统进行问题解答,用户能高效、准确地获取所需信息。文章将深入探讨知识图谱的构建、表示与更新,以及基于知识图谱的问答系统的设计、实现与应用,通过案例分析,揭示其面临的挑战与未来发展方向,并为问答系统的进一步发展提供有益的参考和启示,希望能够给人们获取知识带来更为便捷和智能的体验。

从“第二个结合”看“第二个答案”:习近平总书记关于党的自我革命的重要思想的理论内涵与实践进路

党的十八大以来,习近平总书记带领全党以前所未有的决心与力度推进全面从严治党,创造性提出一系列具有原创性、标志性的新理念新思想新战略,形成了习近平总书记关于党的自我革命的重要思想。这一重要思想是我们党坚持把马克思主义基本原理同中国具体实际相结合、同中华优秀传统文化相结合推进理论创新取得的新成果,是习近平新时代中国特色社会主义思想科学体系的重要组成部分。从中西文明视角考察“革命”到“自我革命”概念史生成源流,“革命”一词本义就体现了“兴也勃焉、亡也忽焉”的历史周期率,与破除旧的政治上层建筑的社会运动,实现新的社会建设运动的西方资产阶级传统“革命”观念不同,“自我革命”承接了马克思主义重构经济结构的内涵,更强调革命主体从自我内因角度出发接受革命性锻造。“党的自我革命”与中华优秀传统文化中蕴含的“革故鼎新”“自省克己”“民为邦本”“正身率下”思想与内涵高度契合,勇于自我革命是中国共产党最鲜明的品格,体现了中国共产党深厚的文化自信。习近平总书记在二十届中央纪委三次全会上发表重要讲话,明确提出“九个以”的实践要求,对持续发力、纵深推进反腐败斗争作出战略部署。这“九个以”的实践要求,既有宏观层面的目标任务、顶层设计,也有落细落实、重点突出的方式方法;既有认识论,又有方法论。我们从“第二个结合”看“第二个答案”,领悟党的自我革命的重要思想理论内涵与实践进路,要从“第二个结合”视角,把握习近平总书记关于党的自我革命的重要思想所蕴含的重大创新观点、科学方法和重要战略部署,更加自觉主动地以伟大自我革命引领伟大社会革命。

一种基于多模态特征提取的医学视觉问答方法

随着深度学习在医疗领域的快速发展,医学视觉问答(Med-VQA)吸引了研究人员的广泛关注.现有的Med-VQA方法大都使用权重参数共享的同一特征提取网络对多模态医学影像进行特征提取,在一定程度上忽略了不同模态医学影像的差异性特征,导致对特定模态特征提取时引入其它模态的噪声特征,使得模型难以关注到不同模态医学影像中的关键特征.针对上述问题,本文提出一种基于多模态特征提取的医学视觉问答方法.首先,对医学影像进行模态识别,根据模态标签指导输入参数不共享的特征提取网络以获得不同模态影像的差异性特征;然后,设计了一种面向Med-VQA的卷积降噪模块以降低医学影像不同模态特征的噪声信息;最后,采用空间与通道注意力模块进一步增强不同模态差异性特征的关注度.在Med-VQA公共数据集Slake上得到的实验结果表明,本文提出方法能有效提高Med-VQA的准确率.

郭店楚简《性自命出》“有为”文学主张的内涵

郭店楚墓竹简中的《性自命出》篇在言及诗、书、礼、乐等传统经典时,分别提出了“有为为之”“有为言之”和“有为举之”的概括。这一概括不仅是对前代文化经典的总结,也蕴含了对文学必须“有为”的理论主张。尤其是其中“有为言之”的说法,更是关涉语言这一文学的核心问题。基于之前礼乐时代人道的思想立场与文化原则,“有为”之学肯定了人文价值的实在性,与道家的虚无主张形成鲜明的对比。而在言语形式上,“有为”的理论主张又最终和礼乐文化相结合,落实到具体的实践规范之上,由此丰富了儒家与先秦文学理论的内涵。

基于层次结构图的多跳知识图谱问答模型

知识图谱问答(KBQA)旨在理解用户的自然语言问句,在结构化的知识图谱中通过检索、推理等手段来获取答案实体。近年来,多跳KBQA备受关注,然而,复杂问句中通常存在多个关系意图,已有KBQA方法大多忽视了推理关系链的关系顺序问题。为此,提出一种基于层次结构图的多跳知识图谱问答模型(HSG-KBQA),建模自然语言问句的关系层次顺序,指导模型在每个推理步选择合理的关系意图。设计一种层次结构图,显式地体现问句中关系的层次距离,利用LSTM-BiGCN编码层将词语间的依存信息编码到问句中;提出虚拟节点的概念,利用图池化技术过滤不重要的节点,学习推理过程中知识图谱的状态;设计基于注意力机制和层次权重的解码器来优化指令生成,使推理指令更匹配问句中的关系链顺序。实验结果表明,HSG-KBQA在WebQuestionsSP数据集上取得了71.3%的Hits@1分数,在PathQuestions数据集上取得了97.3%(PQ-2H)和89.7%(PQ-3H)的Hits@1分数,均优于对照基准模型,表明HSG-KBQA模型在KBQA任务中具有更好的性能。