CSMCCVA:Framework of cross-modal semantic mapping based on cognitive computing of visual and auditory sensations

Cross-modal semantic mapping and cross-media retrieval are key problems of the multimedia search engine.This study analyzes the hierarchy,the functionality,and the structure in the visual and auditory sensations of cognitive system,and establishes a brain-like cross-modal semantic mapping framework based on cognitive computing of visual and auditory sensations.The mechanism of visual-auditory multisensory integration,selective attention in thalamo-cortical,emotional control in limbic system and the memory-enhancing in hippocampal were considered in the framework.Then,the algorithms of cross-modal semantic mapping were given.Experimental results show that the framework can be effectively applied to the cross-modal semantic mapping,and also provides an important significance for brain-like computing of non-von Neumann structure.

Next Generation Semantic and Spatial Joint Perception——Neural Metric-Semantic Understanding

Efficient perception of the real world is a long-standing effort of computer vision.Mod⁃ern visual computing techniques have succeeded in attaching semantic labels to thousands of daily objects and reconstructing dense depth maps of complex scenes.However,simultaneous se⁃mantic and spatial joint perception,so-called dense 3D semantic mapping,estimating the 3D ge⁃ometry of a scene and attaching semantic labels to the geometry,remains a challenging problem that,if solved,would make structured vision understanding and editing more widely accessible.Concurrently,progress in computer vision and machine learning has motivated us to pursue the capability of understanding and digitally reconstructing the surrounding world.Neural metric-se⁃mantic understanding is a new and rapidly emerging field that combines differentiable machine learning techniques with physical knowledge from computer vision,e.g.,the integration of visualinertial simultaneous localization and mapping(SLAM),mesh reconstruction,and semantic un⁃derstanding.In this paper,we attempt to summarize the recent trends and applications of neural metric-semantic understanding.Starting with an overview of the underlying computer vision and machine learning concepts,we discuss critical aspects of such perception approaches.Specifical⁃ly,our emphasis is on fully leveraging the joint semantic and 3D information.Later on,many im⁃portant applications of the perception capability such as novel view synthesis and semantic aug⁃mented reality(AR)contents manipulation are also presented.Finally,we conclude with a dis⁃cussion of the technical implications of the technology under a 5G edge computing scenario.

Adequate alignment and interaction for cross-modal retrieval

Background Cross-modal retrieval has attracted widespread attention in many cross-media similarity search applications,particularly image-text retrieval in the fields of computer vision and natural language processing.Recently,visual and semantic embedding(VSE)learning has shown promising improvements in image text retrieval tasks.Most existing VSE models employ two unrelated encoders to extract features and then use complex methods to contextualize and aggregate these features into holistic embeddings.Despite recent advances,existing approaches still suffer from two limitations:(1)without considering intermediate interactions and adequate alignment between different modalities,these models cannot guarantee the discriminative ability of representations;and(2)existing feature aggregators are susceptible to certain noisy regions,which may lead to unreasonable pooling coefficients and affect the quality of the final aggregated features.Methods To address these challenges,we propose a novel cross-modal retrieval model containing a well-designed alignment module and a novel multimodal fusion encoder that aims to learn the adequate alignment and interaction of aggregated features to effectively bridge the modality gap.Results Experiments on the Microsoft COCO and Flickr30k datasets demonstrated the superiority of our model over state-of-the-art methods.

A Natural Language Generation Algorithm for Greek by Using Hole Semantics and a Systemic Grammatical Formalism

This work is about the progress of previous related work based on an experiment to improve the intelligence of robotic systems,with the aim of achieving more linguistic communication capabilities between humans and robots.In this paper,the authors attempt an algorithmic approach to natural language generation through hole semantics and by applying the OMAS-III computational model as a grammatical formalism.In the original work,a technical language is used,while in the later works,this has been replaced by a limited Greek natural language dictionary.This particular effort was made to give the evolving system the ability to ask questions,as well as the authors developed an initial dialogue system using these techniques.The results show that the use of these techniques the authors apply can give us a more sophisticated dialogue system in the future.

Social network search based on semantic analysis and learning

Because of everyone＇s involvement in social networks, social networks are full of massive multimedia data, and events are got released and disseminated through social networks in the form of multi-modal and multi-attribute heterogeneous data. There have been numerous researches on social network search. Considering the spatio-temporal feature of messages and social relationships among users, we summarized an overall social network search framework from the perspective of semantics based on existing researches. For social network search, the acquisition and representation of spatio-temporal data is the basis, the semantic analysis and modeling of social network cross-media big data is an important component, deep semantic learning of social networks is the key research field, and the indexing and ranking mechanism is the indispensable part. This paper reviews the current studies in these fields, and then main challenges of social network search are given. Finally, we give an outlook to the prospect and further work of social network search.

Exploiting multi-context analysis in semantic image classification

As the popularity of digital images is rapidly increasing on the Internet, research on technologies for semantic image classification has become an important research topic. However, the well-known content-based image classification methods do not overcome the so-called semantic gap problem in which low-level visual features cannot represent the high-level semantic content of images. Image classification using visual and textual information often performs poorly since the extracted textual features are often too limited to accurately represent the images. In this paper, we propose a semantic image classification ap- proach using multi-context analysis. For a given image, we model the relevant textual information as its multi-modal context, and regard the related images connected by hyperlinks as its link context. Two kinds of context analysis models, i.e., cross-modal correlation analysis and link-based correlation model, are used to capture the correlation among different modals of features and the topical dependency among images induced by the link structure. We propose a new collective classification model called relational support vector classifier (RSVC) based on the well-known Support Vector Machines (SVMs) and the link-based cor- relation model. Experiments showed that the proposed approach significantly improved classification accuracy over that of SVM classifiers using visual and/or textual features.

A Fast Panoptic Segmentation Network for Self-Driving Scene Understanding

In recent years,a gain in popularity and significance of science understanding has been observed due to the high paced progress in computer vision techniques and technologies.The primary focus of computer vision based scene understanding is to label each and every pixel in an image as the category of the object it belongs to.So it is required to combine segmentation and detection in a single framework.Recently many successful computer vision methods has been developed to aid scene understanding for a variety of real world application.Scene understanding systems typically involves detection and segmentation of different natural and manmade things.A lot of research has been performed in recent years,mostly with a focus on things(a well-defined objects that has shape,orientations and size)with a less focus on stuff classes(amorphous regions that are unclear and lack a shape,size or other characteristics Stuff region describes many aspects of scene,like type,situation,environment of scene etc.and hence can be very helpful in scene understanding.Existing methods for scene understanding still have to cover a challenging path to cope up with the challenges of computational time,accuracy and robustness for varying level of scene complexity.A robust scene understanding method has to effectively deal with imbalanced distribution of classes,overlapping objects,fuzzy object boundaries and poorly localized objects.The proposed method presents Panoptic Segmentation on Cityscapes Dataset.Mobilenet-V2 is used as a backbone for feature extraction that is pre-trained on ImageNet.MobileNet-V2 with state-of-art encoder-decoder architecture of DeepLabV3+with some customization and optimization is employed Atrous convolution along with Spatial Pyramid Pooling are also utilized in the proposed method to make it more accurate and robust.Very promising and encouraging results have been achieved that indicates the potential of the proposed method for robust scene understanding in a fast and reliable way.

Semantic Holism:An Introduction

This is a paper about semantic holism.Semantic holism is a doctrine about the philosophically indispensable circumstances for something to have meaning.We therefore instigate our argument by endeavoring to view semantic holism in its philosophical context.The crucial contention of this paper is to resolve some fundamental issues like semantic similarity and dissimilarity,semantic atomism,holism,etc.It is also tries to resolve some issues related to the nature and scope of semantics.

语言学知识驱动的空间语义理解能力评测数据集研究

近20年来,深度学习技术显著提升了机器的自然语言处理能力,使之在诸多任务上接近甚至超过人类水平。机器学习的对象不再是直接来自人类语言学研究成果(知识),而是人类语言材料(数据)。在靠数据和算力驱动的大语言模型几近建成巴别塔的当下,语言学家通过深挖语言现象总结的语言学知识价值何在?本文提出从知识到数据的研究思路,设计了空间语义理解的6项任务:空间信息正误判别、异常空间信息识别、缺失参照成分补回、空间语义角色标注、空间表达异形同义判别、空间方位关系推理,以构建中文空间语义理解能力评测数据集为例,介绍从SpaCE2021到SpaCE2024数据集的设计思想、数据集制作概况以及机器在空间语义理解任务上的表现。总的来看,参加SpaCE赛事的大语言模型,在依赖表面分布特征(形式线索)的任务上容易获得好成绩,在依赖深层语义理解(认知能力)的任务上容易表现不好。因此,在人工智能高速发展使得语言学知识在计算机信息处理领域被动边缘化的当下,语言学知识的价值需要拓展,即用于指导小而精的高品质语言数据,以提升机器学习的效果和效率。为了计算应用的目的,语法研究应该在观察充分、描写充分、解释充分之上,追求更具挑战性的目标——生成充分。

如何测试ChatGPT的语义理解与常识推理水平?——兼谈大语言模型时代语言学的挑战与机会

ChatGPT等大语言模型在语义理解和常识推理方面表现优秀,其技术奥秘在于模型开发者在对词语进行向量表示时,遵循分布式语义学原理,采用了“嵌入”这种代数方法。但是,经典的测试语言运用等智能水平的“图灵测试”,难以识别欺骗和回避等作弊手段,因此,计算机科学家设计了“新图灵测试”,其中的威诺格拉德模式挑战与语言学关系密切。这一模式以代词消歧为测试点设计句子对和问题,但是经过大规模语料训练的语言模型可以凭借词汇上的统计相关性,而不是靠真正理解句子的意义来给出正确答案。为克服这一缺陷,学者们又发展出WinoGrande数据集,提高了数据的规模和难度,确保它们无法通过网络搜索等手段来得到正确答案。我们用威诺格拉德模式设计了无偏向双重句子对测试ChatGPT,展示了大语言模型在语义理解和常识推理方面已达到接近人类的水平。当然,从具身模拟假说来看,大语言模型不可能像人一样富有体验性地理解人类自然语言。最后,我们呼吁:语言学家应该积极参与构建WinoGrade测试集之类的工作,在人工智能时代扩展自身的学术研究领域。

二序理解在语意学转向下的设计意义

利益相关者对人工物往往有着自身独特的理解,为了使设计师能够对这些理解进行理解,就需要将二序理解融入设计师的思维之中。关于设计中的二序理解的研究离不开语意学转向的支持,也同样需要语义学理论的支持。基于以人为本的设计与设计的语意学转向,文章在利益相关者对彼此以及人工物的理解中展开对二序理解的研究,得到了二序理解在设计中的实现方式,进而讨论了二序理解在某些维度上对设计的意义。

基于多尺度时空Transformer的视频动态场景图生成模型

为应对动态视频中物体间关系在时间维度上的动态变化,提出一种基于多尺度时空Transformer的视频动态场景图生成模型,在经典的Transformer架构基础上引入了多尺度建模思想,以实现对视频动态细粒度语义的精确建模。首先,在空间维度上保留了传统模型对物体在全局空间相关性的关注;同时还对物体间的相对位置进行了局部空间相关性建模,以便更好地理解人和物之间的交互动态,提供更准确的语义分析结果。其次,在时间维度上,除了保留传统模型对视频中物体短期时间相关性的关注外,还关注了同一对物体在完整视频中的长期时间相关性,通过更全面地建模物体之间的长期关系,生成更准确、连贯的场景图,在一定程度上缓解了由遮挡、重合等引起的场景图生成问题。最后,通过空间编码器与时间编码器的共同作用,更加精准地建模视频动态细粒度语义,克服了传统的单尺度模型的局限性。实验结果显示,在Action Genome基准数据集上,与基线模型STTran相比,在谓词分类、场景图分类与场景图检测三个任务的Recall@10指标上分别提升了5.0、2.8、2.9个百分点。实验结果表明,多尺度建模思想能够更加精确地建模,并有效地提高在视频动态场景图生成任务上的性能。

融合语义理解的航站楼显示设备故障检测方法

针对航站楼内显示设备出现的故障画面,提出一种融合语义理解的故障检测方法。首先设计滚动文字拼接技术提取界面文字信息;然后根据机场业务背景融入语义规则,训练得到故障分类模型,实现对显示设备出现的非正常显示界面和显示信息歧义等故障的智能检测;最后使用神经网络模型压缩技术将模型轻量化并部署在SOM-RK3399嵌入式设备上。实验表明,融合语义理解模块的检测方法的分类准确率达到88.74%,该方法能够有效解决传统故障检测技术的不足,提高故障检测效率,减少人工检验情况。

一种辅助决策智能体及与生成式AI的联合应用

现阶段,受限于数据的私域属性、数据治理品质的差异,垂直领域细分场景较高的复杂机理和专业知识门槛,以及较高的后服务成本,面向通识应用的生成式人工智能大模型或经小规模数据微调的大模型,仍难以在工业制造、能源生产、资产管理等场景下,解决用户希望结合现象,即时获得相应的推理决策结论,并实现一定程度的自主持续学习提升的需求。为此,该文提出了一种适用油气行业的自监督辅助决策智能体系统,基于多个不同角色智能体间的反馈和自监督,在问答决策类场景中,弥补大模型在专业语义理解、多轮自交互、判断决策及自主优化方面的短板,同时探索应用中等规模预训练大模型,面向垂直场景,实现类似超大规模基座模型表现的可行性。通过在钻井井控、装置资产运维管理、炼化装置操作指引三个高频场景中与AIGC大模型的联合应用,论证了该方法对降低大模型的工程应用门槛、可解释、降低应用成本等方面的有效性。

基于语义理解的短视频一键成片系统设计与实现

介绍一个基于语义理解的一键成片系统,为市场营销和媒体短视频制作提供高效的短视频生成解决方案。对于市场营销类场景,系统能够快速生成风格统一、信息准确且视觉效果突出的产品宣传视频,以满足大规模的营销需求。在媒体类应用中,系统侧重于视频质量,通过语义理解生成脚本,并结合口播驱动的方式匹配相关视频素材,进而生成高质量的混剪短视频。该系统利用人工智能(Artificial Intelligence,AI)有效提高短视频的生产效率,降低了人力成本,激发视频制作的创意潜能。

基于自然语言处理的铁路客运营销分析智能对话系统研究

为提高铁路客运营销数据分析能力,研究开发了铁路客运营销分析智能对话系统,为铁路客运营销业务人员提供一种基于人机对话的数据分析工具。该系统包括语音识别、自然语言文本处理、智能数据挖掘、智能应答4个主要功能模块;利用语音唤醒和语音识别技术采集语音数据,通过神经网络模型将语音数据转换成自然语言文本;建立自然语言文本预处理模型,完成基于规则的词法句法分析方法,使用长短期记忆神经网络实现语义理解,确定用户意图;基于Bert模型的Text-to-SQL技术,将自然语言文本数据转换成数据查询SQL语句,构建智能Agent完成数据挖掘分析,生成数据分析结果;最后,运用语音合成技术和数据可视化技术,将数据分析结果转换为用户应答信息。

人工智能关键技术在电力客户服务中的应用

现有的智能客服系统具有一定的局限性,通常仅限于处理简单的用户查询,难以理解和应对复杂的用户表达。为了提升多轮对话中的语义理解准确性,深入研究了电力智能客服的应用框架,并将语言理解作为核心研究领域;提出了一种基于融合卷积神经网络(CNN)和注意力机制的语言理解模型,并成功将其应用于电力智能客服系统中。实验结果表明,这一融合了CNN和注意力机制的语言理解模型在语义理解任务中表现出色,能够更准确地捕捉用户的意图,从而显著提升了智能客服系统的智能程度和可用性。

政务智能问答系统评价指标体系构建与测评问题编制

[目的/意义]智能问答系统正成为提供公共服务的重要设施。通过评价可以促进政务智能问答系统建设,降低用户行政负担。[研究设计/方法]基于文献调研、理论分析和案例研究,构建政务智能问答系统评价指标体系,并运用专家调查和AHP法确定了指标权重;根据政务热点咨询问题编制了与问题解决质量维度配套的测试问题集;对我国30个省级政府网站智能问答系统进行了实际测评。[结论/发现]政务智能问答系统评价指标体系包含问题解决质量、服务交互质量、基础建设质量3个维度以及高频问题解答率、同义问题解答率、交互次数等18个细化指标;测试题集包括高频、同义、错误、省略、英文五大类问题。实际评价证明了指标体系和测试题集的可用性和有效性。[创新/价值]从降低行政负担视角出发构建政务智能问答系统评价指标体系,运用文本挖掘技术构建测评问题集,并通过实际测评证明其可用性。

基于语义理解力的我国省级政府网站智能问答服务质量评价研究

[目的/意义]智能问答系统已成为各类网站提供信息咨询服务的重要设施。政务咨询问题的复杂性,对政府网站智能问答系统的语义理解能力提出了更高要求。[方法/过程]文章采用南开大学网络社会治理研究中心开发的“基于语义理解力的政府网站智能客服评价体系”以及配套的测试题集,对我国30个省级政府网站智能问答系统的“问题解决质量”“服务交互质量”“基础建设质量”进行了评测。[结果/结论]上海、浙江、北京总分名列前三;当前政府网站智能问答系统在语义理解及场景化服务方面存在明显不足,仅有30%的系统得分达到及格线以上,部分问答系统在基础功能与服务交互方面存在较大提升空间。最后,文章提出扩展知识库、提升问题匹配精度、增加人文关怀等对策建议。

基于ViT与语义引导的视频内容描述生成

现有视频内容描述模型生成的视频内容描述文本可读性差且准确率不高。基于ViT模型提出一种语义引导的视频内容描述方法。利用ReNeXt和ECO网络提取视频的视觉特征,以提取的视觉特征为输入、语义标签的概率预测值为输出训练语义检测网络(SDN)。在此基础上,通过ViT模型对静态和动态视觉特征进行全局编码,并与SDN提取的语义特征进行注意力融合,采用语义长短期记忆网络对融合特征进行解码,生成视频对应的描述文本。通过引入视频中的语义特征能够引导模型生成更符合人类习惯的描述,使生成的描述更具可读性。在MSR-VTT数据集上的测试结果表明,该模型的BLEU-4、METEOR、ROUGE-L和CIDEr指标分别为44.8、28.9、62.8和51.1,相比于当前主流的视频内容描述模型ADL和SBAT,提升的得分总和达到16.6和16.8。