Deep Multi-Module Based Language Priors Mitigation Model for Visual Question Answering

The original intention of visual question answering(VQA)models is to infer the answer based on the relevant information of the question text in the visual image,but many VQA models often yield answers that are biased by some prior knowledge,especially the language priors.This paper proposes a mitigation model called language priors mitigation-VQA(LPM-VQA)for the language priors problem in VQA model,which divides language priors into positive and negative language priors.Different network branches are used to capture and process the different priors to achieve the purpose of mitigating language priors.A dynamically-changing language prior feedback objective function is designed with the intermediate results of some modules in the VQA model.The weight of the loss value for each answer is dynamically set according to the strength of its language priors to balance its proportion in the total VQA loss to further mitigate the language priors.This model does not depend on the baseline VQA architectures and can be configured like a plug-in to improve the performance of the model over most existing VQA models.The experimental results show that the proposed model is general and effective,achieving state-of-the-art accuracy in the VQA-CP v2 dataset.

PAL-BERT:An Improved Question Answering Model

In the field of natural language processing(NLP),there have been various pre-training language models in recent years,with question answering systems gaining significant attention.However,as algorithms,data,and computing power advance,the issue of increasingly larger models and a growing number of parameters has surfaced.Consequently,model training has become more costly and less efficient.To enhance the efficiency and accuracy of the training process while reducing themodel volume,this paper proposes a first-order pruningmodel PAL-BERT based on the ALBERT model according to the characteristics of question-answering(QA)system and language model.Firstly,a first-order network pruning method based on the ALBERT model is designed,and the PAL-BERT model is formed.Then,the parameter optimization strategy of the PAL-BERT model is formulated,and the Mish function was used as an activation function instead of ReLU to improve the performance.Finally,after comparison experiments with traditional deep learning models TextCNN and BiLSTM,it is confirmed that PALBERT is a pruning model compression method that can significantly reduce training time and optimize training efficiency.Compared with traditional models,PAL-BERT significantly improves the NLP task’s performance.

DPAL-BERT:A Faster and Lighter Question Answering Model

Recent advancements in natural language processing have given rise to numerous pre-training language models in question-answering systems.However,with the constant evolution of algorithms,data,and computing power,the increasing size and complexity of these models have led to increased training costs and reduced efficiency.This study aims to minimize the inference time of such models while maintaining computational performance.It also proposes a novel Distillation model for PAL-BERT(DPAL-BERT),specifically,employs knowledge distillation,using the PAL-BERT model as the teacher model to train two student models:DPAL-BERT-Bi and DPAL-BERTC.This research enhances the dataset through techniques such as masking,replacement,and n-gram sampling to optimize knowledge transfer.The experimental results showed that the distilled models greatly outperform models trained from scratch.In addition,although the distilled models exhibit a slight decrease in performance compared to PAL-BERT,they significantly reduce inference time to just 0.25%of the original.This demonstrates the effectiveness of the proposed approach in balancing model performance and efficiency.

A survey of deep learning-based visual question answering

With the warming up and continuous development of machine learning,especially deep learning,the research on visual question answering field has made significant progress,with important theoretical research significance and practical application value.Therefore,it is necessary to summarize the current research and provide some reference for researchers in this field.This article conducted a detailed and in-depth analysis and summarized of relevant research and typical methods of visual question answering field.First,relevant background knowledge about VQA(Visual Question Answering)was introduced.Secondly,the issues and challenges of visual question answering were discussed,and at the same time,some promising discussion on the particular methodologies was given.Thirdly,the key sub-problems affecting visual question answering were summarized and analyzed.Then,the current commonly used data sets and evaluation indicators were summarized.Next,in view of the popular algorithms and models in VQA research,comparison of the algorithms and models was summarized and listed.Finally,the future development trend and conclusion of visual question answering were prospected.

ALBERT with Knowledge Graph Encoder Utilizing Semantic Similarity for Commonsense Question Answering

Recently,pre-trained language representation models such as bidirec-tional encoder representations from transformers(BERT)have been performing well in commonsense question answering(CSQA).However,there is a problem that the models do not directly use explicit information of knowledge sources existing outside.To augment this,additional methods such as knowledge-aware graph network(KagNet)and multi-hop graph relation network(MHGRN)have been proposed.In this study,we propose to use the latest pre-trained language model a lite bidirectional encoder representations from transformers(ALBERT)with knowledge graph information extraction technique.We also propose to applying the novel method,schema graph expansion to recent language models.Then,we analyze the effect of applying knowledge graph-based knowledge extraction techniques to recent pre-trained language models and confirm that schema graph expansion is effective in some extent.Furthermore,we show that our proposed model can achieve better performance than existing KagNet and MHGRN models in CommonsenseQA dataset.

A novel approach for agent ontology and its application in question answering

The information integration method of semantic web based on agent ontology(SWAO method) was put forward aiming at the problems in current network environment,which integrates,analyzes and processes enormous web information and extracts answers on the basis of semantics. With SWAO method as the clue,the following technologies were studied:the method of concept extraction based on semantic term mining,agent ontology construction method on account of multi-points and the answer extraction in view of semantic inference. Meanwhile,the structural model of the question answering system applying ontology was presented,which adopts OWL language to describe domain knowledge from where QA system infers and extracts answers by Jena inference engine. In the system testing,the precision rate reaches 86%,and the recalling rate is 93%. The experimental results prove that it is feasible to use the method to develop a question answering system,which is valuable for further study in more depth.

Information Extraction Based on Multi-turn Question Answering for Analyzing Korean Research Trends

Analyzing Research and Development(R&D)trends is important because it can influence future decisions regarding R&D direction.In typical trend analysis,topic or technology taxonomies are employed to compute the popularities of the topics or codes over time.Although it is simple and effective,the taxonomies are difficult to manage because new technologies are introduced rapidly.Therefore,recent studies exploit deep learning to extract pre-defined targets such as problems and solutions.Based on the recent advances in question answering(QA)using deep learning,we adopt a multi-turn QA model to extract problems and solutions from Korean R&D reports.With the previous research,we use the reports directly and analyze the difficulties in handling them using QA style on Information Extraction(IE)for sentence-level benchmark dataset.After investigating the characteristics of Korean R&D,we propose a model to deal with multiple and repeated appearances of targets in the reports.Accordingly,we propose a model that includes an algorithm with two novel modules and a prompt.A newly proposed methodology focuses on reformulating a question without a static template or pre-defined knowledge.We show the effectiveness of the proposed model using a Korean R&D report dataset that we constructed and presented an in-depth analysis of the benefits of the multi-turn QA model.

Application of Question and Answering on Virtual Human Dialogue:a Review and Prediction

Nowadays,virtual human(VH) is becoming a hot research topic in virtualization.VH dialogue can be categorized as an application of natural language processing(NLP) technology,since it is relational to question and answering(QA) technologies.In order to integrate these technologies,this paper reviews some important work on VH dialogue,and predicts some research points on the view of QA technologies.

Prompting Large Language Models with Knowledge-Injection for Knowledge-Based Visual Question Answering

Previous works employ the Large Language Model(LLM)like GPT-3 for knowledge-based Visual Question Answering(VQA).We argue that the inferential capacity of LLM can be enhanced through knowledge injection.Although methods that utilize knowledge graphs to enhance LLM have been explored in various tasks,they may have some limitations,such as the possibility of not being able to retrieve the required knowledge.In this paper,we introduce a novel framework for knowledge-based VQA titled“Prompting Large Language Models with Knowledge-Injection”(PLLMKI).We use vanilla VQA model to inspire the LLM and further enhance the LLM with knowledge injection.Unlike earlier approaches,we adopt the LLM for knowledge enhancement instead of relying on knowledge graphs.Furthermore,we leverage open LLMs,incurring no additional costs.In comparison to existing baselines,our approach exhibits the accuracy improvement of over 1.3 and 1.7 on two knowledge-based VQA datasets,namely OK-VQA and A-OKVQA,respectively.

面向私有问答系统的检索增强式大模型稳定输出方法

基于大模型的问答系统受大模型语义不一致性问题的影响,会出现“输出结果不稳定”的现象,从而制约着问答系统的安全性、鲁棒性和可信度,严重影响了用户体验。针对上述问题,提出一种面向私有问答系统的检索增强式大模型稳定输出方法。该方法通过优化提示词,让大模型首先输出num_k个用户查询的同义查询,然后输出答案;目的是在大模型输出答案时,可以参考已经输出的num_k个同义查询,从而使大模型的输出结果更加稳定。针对开源大模型因指令理解能力弱而出现的“同义查询生成数目不稳定、输出格式无法解析”等问题,提出通过数据蒸馏的方式,利用闭源大模型自动构建了一个开放域上的检索增强式指令数据集,在该指令集上对开源大模型进行微调。构建了一个私有问答场景下的评估集以验证该方法的有效性。在上述评估集上的实验结果表明,该方法在一致性指标和效果指标上,均显著优于基线方法。与基线方法相比,该方法的一致性指标ROUGE-1、ROUGE-2、ROUGE-L和BLEU分别提升了18.9、30.1、24.5和30.6个百分点,效果指标正确率提升了17.4个百分点。

大语言模型融合知识图谱与向量检索的问答系统

随着深度学习的发展,大型神经网络在自然语言处理领域得到广泛应用。然而,基于大模型的问答系统存在幻觉、失效过期等问题,且难以捕捉实体之间的复杂关系,导致结果偏差。鉴于此,提出一种利用大语言模型微调构建知识图谱和向量检索的融合问答系统。系统通过微调大模型实现知识图谱构建与应用、多模型混合调用;结合知识图谱搜索和向量搜索实现检索结果优化。系统包括图查询模型微调、知识图谱抽取模型微调、知识图谱与向量数据库构建、融合检索与排序4个模块。图查询模型和知识图谱抽取模型分别用于生成图查询语句和抽取三元组知识;知识图谱存储在Neo4j中,文本向量存储在向量数据库中(postgres vector,PGVector)中;融合检索综合利用知识图谱和向量搜索结果。结果表明:在标准问答数据集(the stanford question answering dataset,SQuAD)上,融合检索方法的F1值为0.77,优于单一的知识图谱检索(0.73)和向量检索(0.74)方法。专家评估也表明,融合方法的结果最佳。该融合问答系统能充分发挥大模型、知识图谱和向量检索的优势,提高了问答的准确性和全面性。未来可在知识图谱更新、模型偏见减少和系统优化等方面开展进一步研究。

基于大语言模型的Linux课程问答系统

基于国产主流大语言模型,设计一个Linux课程知识问答系统.该系统结合检索增强技术,能根据人类反馈持续学习,有助于解决Linux课程教学中如何更有效辅助学生学习的问题.实验结果表明,该系统提高了大语言模型回答的事实性,能有效回答学生提问.此外,该系统以较低成本积累了以自然语言形式呈现的专业领域知识库,降低了教师教学资料搜集整理的工作量.

大语言模型辅助下的增强现实装配方法

基于增强现实的装配引导系统将数字信息叠加到物理场景中,有效指导了复杂装配作业任务。然而装配环境中人与物理世界的隔阂仍然巨大,待融合到物理世界的信息需事先准备好,并且需要人工在装配过程中来触发。研究实时且无处不在的提示,成为当前增强现实环境下的复杂装配研究热点,本文提出了一种基于大语言模型(LLMs)辅助的增强现实装配方法,其核心是将LLMs作为装配过程中的另外一个大脑,提供无处不在的装配引导和工艺信息提示支持。首先,建立了LLMs辅助的增强现实装配方法体系,分析了体系的要素及关联关系。其次,面向LLMs环境,构建了匹配的工艺信息模型。随后,给出了基于LLMs的辅助引导装配方法和流程。最后,结合某线缆装配专业知识,研发了一个专业问答系统,实现了LLMs智能辅助引导,使装配合格率提升了15%,并通过多个案例验证了该方法的有效性。

大语言模型中文问答正确性对比实验研究——以ChatGPT 3.5、Claude 1.0和文心一言2.1为例

[目的/意义]对大语言模型中文问答正确性进行实验测评研究,为中文用户使用大语言模型提供一定的指导作用。[方法/过程]针对科技、教育、医学、生活、旅游美食和哲学文化6个领域,分别设计常识性、专业性和开放性三类问题,每类20个问题,共计360个问题。分别向ChatGPT 3.5、Claude 1.0和文心一言2.1提问,再针对回答进行正确性的人工评价。最后汇总评价结果,进行正确性的多方面对比分析。[结果/结论]实验分析表明中文语料数据的规模与质量,以及大语言模型的参数规模是影响大语言模型中文问答正确性的重要因素。

基于大语言模型的智能问答系统在高校中的设计与应用

随着信息技术的迅速发展,智能问答系统已经成为高等教育领域提高教学效率和资源可及性的重要工具。文中探讨了基于大语言模型(LLM)的智能问答系统在高校中的设计与应用。通过分析当前智能问答系统的发展现状和面临的挑战,提出了一个整合了最新自然语言处理技术和人工智能的智能问答系统框架。该系统旨在为学生、教师及其他利益相关者提供快速准确的信息检索和问答服务,涵盖了招生信息、奖学金资讯、职业发展指导、学术辅导等多个方面。通过案例分析,文中展示了系统的设计过程、关键技术、实现挑战及解决策略,旨在为高等教育机构实施智能问答系统提供参考。

一种消减多模态偏见的鲁棒视觉问答方法

为了增强视觉问答模型的鲁棒性,提出一种偏见消减方法,并在此基础上探究语言与视觉信息对偏见的影响。进一步地,构造两个偏见学习分支来分别捕获语言偏见以及语言和图片共同导致的偏见,利用偏见消减方法,得到鲁棒性更强的预测结果。最后,依据标准视觉问答与偏见分支之间的预测概率差异,对样本进行动态赋权,使模型针对不同偏见程度的样本动态地调节学习程度。在VQA-CP v2.0等数据集上的实验结果证明了所提方法的有效性,缓解了偏见对模型的影响。

长江流域取水许可知识图谱问答系统

随着水资源取水许可领域管理要求的不断提高,传统水资源取水许可信息管理系统难以满足复杂的信息检索需求,制约了水资源精细化管理水平的提升。为了打破系统间信息孤岛,提升取水许可信息检索效率,建立了长江流域取水许可知识图谱,基于大规模预训练语言模型提出了包含实体提及识别、实体链接、关系匹配等功能的知识图谱问答流水线方法,结合取水许可领域数据特点采用BM25算法进行候选实体排序,构建了长江流域取水许可知识图谱问答系统,并基于BS架构开发了Web客户端。实验表明:该系统在测试集上达到了90.37%的准确率,可支撑长江流域取水许可领域检索需求。

表格问答研究综述

表格问答通过自然语言问句直接与表格数据进行交互并得到答案,是智能问答的主要形式之一。近年来,研究人员利用以语义解析为主的技术在该领域开展了深入研究。该文从不同表格类型分类及其问答任务问题定义出发,将表格问答细分为单表单轮、多表单轮、多表多轮式问答三种任务,并系统介绍了各类表格问答任务的数据集及其代表性方法。其次,该文总结了当前主流表格预训练模型的数据构造、输入编码以及预训练目标。最后,探讨当前工作的优势与不足,并分析了未来表格问答的前景与挑战。

大语言模型增强的知识图谱问答研究进展综述

知识图谱问答(knowledge graph question answering,KGQA)是一种通过处理用户提出的自然语言问题,从知识图谱中获取相关答案的技术。早期的知识图谱问答技术受到知识图谱规模、计算能力以及自然语言处理能力的限制,准确率较低。近年来,随着人工智能技术的进步,特别是大语言模型(large language model,LLM)的发展,知识图谱问答技术的性能得到显著提升。大语言模型如GPT-3等已经被广泛应用于增强知识图谱问答的性能。为了更好地研究学习增强知识图谱问答的技术,对现有的各种大语言模型增强的知识图谱问答方法进行了归纳分析。总结了大语言模型和知识图谱问答的相关知识,即大语言模型的技术原理、训练方法,以及知识图谱、问答和知识图谱问答的基本概念。从语义解析和信息检索两个维度,综述了大语言模型增强知识图谱问答的现有方法,分析了方法所解决的问题及其局限性。收集整理了大语言模型增强知识图谱问答的相关资源和评测方法,并对现有方法的性能表现进行了总结。最后针对现有方法的局限性,分析并提出了未来的重点研究方向。

问答式林业预训练语言模型ForestBERT

【目的】针对林业文本利用率低、通用领域预训练语言模型对林业知识理解不足以及手动标注数据耗时费力等问题,基于大量林业文本,提出一种融合林业领域知识的预训练语言模型,并通过自动标注训练数据,高效实现林业抽取式问答,为林业决策管理提供智能化信息服务。【方法】首先,基于网络爬虫技术构建包含术语、法律法规和文献3个主题的林业语料库,使用该语料库对通用领域预训练语言模型BERT进行继续预训练,再通过掩码语言模型和下一句预测这2个任务进行自监督学习,使BERT能够有效地学习林业语义信息,得到具有林业文本通用特征的预训练语言模型ForestBERT。然后,对预训练语言模型mT5进行微调,实现样本的自动标注,通过人工校正后,构建包含3个主题共2280个样本的林业抽取式问答数据集。基于该数据集对BERT、RoBERTa、MacBERT、PERT、ELECTRA、LERT 6个通用领域的中文预训练语言模型以及本研究构建的ForestBERT进行训练和验证,以明确ForestBERT的优势。为探究不同主题对模型性能的影响,分别基于林业术语、林业法律法规、林业文献3个主题数据集对所有模型进行微调。将ForestBERT与BERT在林业文献中的问答结果进行可视化比较,以更直观展现ForestBERT的优势。【结果】ForestBERT在林业领域的抽取式问答任务中整体表现优于其他6个对比模型,与基础模型BERT相比,精确匹配(EM)分数和F1分数分别提升1.6%和1.72%,在另外5个模型的平均性能上也均提升0.96%。在各个模型最优划分比例下,ForestBERT在EM上分别优于BERT和其他5个模型2.12%和1.2%,在F1上分别优于1.88%和1.26%。此外,ForestBERT在3个林业主题上也均表现优异,术语、法律法规、文献任务的评估分数分别比其他6个模型平均提升3.06%、1.73%、2.76%。在所有模型中,术语任务表现最佳,F1的平均值达到87.63%,表现较差的法律法规也达到82.32%。在文献抽取式问答任务中,ForestBERT相比BERT可提供更准确、全面的答案。【结论】采用继续预训练的方式增强通用领域预训练语言模型的林业专业知识,可有效提升模型在林业抽取式问答任务中的表现,为林业文本和其他领域的文本处理和应用提供一种新思路。