Background In this study,we propose a novel 3D scene graph prediction approach for scene understanding from point clouds.Methods It can automatically organize the entities of a scene in a graph,where objects are nodes...Background In this study,we propose a novel 3D scene graph prediction approach for scene understanding from point clouds.Methods It can automatically organize the entities of a scene in a graph,where objects are nodes and their relationships are modeled as edges.More specifically,we employ the DGCNN to capture the features of objects and their relationships in the scene.A Graph Attention Network(GAT)is introduced to exploit latent features obtained from the initial estimation to further refine the object arrangement in the graph structure.A one loss function modified from cross entropy with a variable weight is proposed to solve the multi-category problem in the prediction of object and predicate.Results Experiments reveal that the proposed approach performs favorably against the state-of-the-art methods in terms of predicate classification and relationship prediction and achieves comparable performance on object classification prediction.Conclusions The 3D scene graph prediction approach can form an abstract description of the scene space from point clouds.展开更多
Scene graph is a infrastructure of the virtual reality system to organize the virtual scene with abstraction, it can provide facility for the rendering engine and should be integrated effectively on demand into a real...Scene graph is a infrastructure of the virtual reality system to organize the virtual scene with abstraction, it can provide facility for the rendering engine and should be integrated effectively on demand into a real-time system, where a large quantities of scene objects and resources can be manipulated and managed with high flexibility and reliability. We present a new scheme of multiple scene graphs to accommodate the features of rendering engine and distributed systems. Based upon that, some other functions, e.g. block query, interactive editing, permission management, instance response, "redo" and "undo", are implemented to satisfy various requirements. At the same time, our design has compatibility to popular C/S architecture with good concurrent performance. Above all, it is convenient to be used for further development. The results of experiments including responding time demonstrate its good performance.展开更多
Traffic scene captioning technology automatically generates one or more sentences to describe the content of traffic scenes by analyzing the content of the input traffic scene images,ensuring road safety while providi...Traffic scene captioning technology automatically generates one or more sentences to describe the content of traffic scenes by analyzing the content of the input traffic scene images,ensuring road safety while providing an important decision-making function for sustainable transportation.In order to provide a comprehensive and reasonable description of complex traffic scenes,a traffic scene semantic captioningmodel withmulti-stage feature enhancement is proposed in this paper.In general,the model follows an encoder-decoder structure.First,multilevel granularity visual features are used for feature enhancement during the encoding process,which enables the model to learn more detailed content in the traffic scene image.Second,the scene knowledge graph is applied to the decoding process,and the semantic features provided by the scene knowledge graph are used to enhance the features learned by the decoder again,so that themodel can learn the attributes of objects in the traffic scene and the relationships between objects to generate more reasonable captions.This paper reports extensive experiments on the challenging MS-COCO dataset,evaluated by five standard automatic evaluation metrics,and the results show that the proposed model has improved significantly in all metrics compared with the state-of-the-art methods,especially achieving a score of 129.0 on the CIDEr-D evaluation metric,which also indicates that the proposed model can effectively provide a more reasonable and comprehensive description of the traffic scene.展开更多
Scene graphs of point clouds help to understand object-level relationships in the 3D space.Most graph generation methods work on 2D structured data,which cannot be used for the 3D unstructured point cloud data.Existin...Scene graphs of point clouds help to understand object-level relationships in the 3D space.Most graph generation methods work on 2D structured data,which cannot be used for the 3D unstructured point cloud data.Existing point-cloud-based methods generate the scene graph with an additional graph structure that needs labor-intensive manual annotation.To address these problems,we explore a method to convert the point clouds into structured data and generate graphs without given structures.Specifically,we cluster points with similar augmented features into groups and establish their relationships,resulting in an initial structural representation of the point cloud.Besides,we propose a Dynamic Graph Generation Network(DGGN)to judge the semantic labels of targets of different granularity.It dynamically splits and merges point groups,resulting in a scene graph with high precision.Experiments show that our methods outperform other baseline methods.They output reliable graphs describing the object-level relationships without additional manual labeled data.展开更多
Image paragraph generation aims to generate a long description composed of multiple sentences,which is different from traditional image captioning containing only one sentence.Most of previous methods are dedicated to...Image paragraph generation aims to generate a long description composed of multiple sentences,which is different from traditional image captioning containing only one sentence.Most of previous methods are dedicated to extracting rich features from image regions,and ignore modelling the visual relationships.In this paper,we propose a novel method to generate a paragraph by modelling visual relationships comprehensively.First,we parse an image into a scene graph,where each node represents a specific object and each edge denotes the relationship between two objects.Second,we enrich the object features by implicitly encoding visual relationships through a graph convolutional network(GCN).We further explore high-order relations between different relation features using another graph convolutional network.In addition,we obtain the linguistic features by projecting the predicted object labels and their relationships into a semantic embedding space.With these features,we present an attention-based topic generation network to select relevant features and produce a set of topic vectors,which are then utilized to generate multiple sentences.We evaluate the proposed method on the Stanford image-paragraph dataset which is currently the only available dataset for image paragraph generation,and our method achieves competitive performance in comparison with other state-of-the-art(SOTA)methods.展开更多
对现实中的树木结构进行研究,在L系统的基础上对字符集进行扩展,设计了更加丰富的树木L文法描述,提出了比较符合树木生长过程的几何幂衰减规律,并结合图形渲染引擎OSG(open scene graph)实现了三维树木的树干部分的绘制。通过给树木的...对现实中的树木结构进行研究,在L系统的基础上对字符集进行扩展,设计了更加丰富的树木L文法描述,提出了比较符合树木生长过程的几何幂衰减规律,并结合图形渲染引擎OSG(open scene graph)实现了三维树木的树干部分的绘制。通过给树木的树干部分添加纹理、绘制树叶并给树叶添加阴影,使得绘制的树木更加真实自然。将绘制的树木作为一个节点加载到虚拟场景中,增强了虚拟场景的真实感。展开更多
基于osg Earth开源工程是将地理信息系统和虚拟现实技术结合起来的一个重要应用。以基于osg Earth三维数字地球建设的研究为例,采用LOD与分页、动态调度等技术。研究了三维地形、影像、地物组织、调度及融合方式;设计了地球影像数据获...基于osg Earth开源工程是将地理信息系统和虚拟现实技术结合起来的一个重要应用。以基于osg Earth三维数字地球建设的研究为例,采用LOD与分页、动态调度等技术。研究了三维地形、影像、地物组织、调度及融合方式;设计了地球影像数据获取、坐标纹理校正、earth文件编写、矢量数据栅格化显示;分析了osg Earth中绘制线状矢量数据的模板体技术和该技术所隐藏的场景层次结构。在Visual Studio 2010平台下,使用Open Scene Graph语言进行编程,证明了该技术的可行性,实现了三维数字地球的建设。展开更多
针对无人机飞行控制仿真环境以数据方式输出仿真结果不利于挖掘隐藏于数据中的本质特性的缺陷,借助虚拟现实技术的相关理论,设计实现基于无人机飞行控制仿真环境的三维视景子系统,实时动态再现无人机飞行运动全过程,为研究无人机飞行运...针对无人机飞行控制仿真环境以数据方式输出仿真结果不利于挖掘隐藏于数据中的本质特性的缺陷,借助虚拟现实技术的相关理论,设计实现基于无人机飞行控制仿真环境的三维视景子系统,实时动态再现无人机飞行运动全过程,为研究无人机飞行运动规律提供直观的事实根据。阐述了无人机飞行控制仿真环境三维视景子系统设计实现的目标,构造了系统结构并提出了三维视景系统的实现框架,对视景系统实现的主要技术进行了着重介绍,在视景系统的具体实现中对视景数据库设计、视景系统开发、视点切换、LynX Prim e和ACF配置文件、数据通信接口的具体实现进行了详细的论述。最后给出了系统的实施效果。展开更多
An ultra-massive distributed virtual environment generally consists of ultra-massive terrain data and a large quantity of objects and their attribute data, such as 2D/3D geometric models, audio/video, images, vectors,...An ultra-massive distributed virtual environment generally consists of ultra-massive terrain data and a large quantity of objects and their attribute data, such as 2D/3D geometric models, audio/video, images, vectors, characteristics, etc. In this paper, we propose a novel method for constructing distributed scene graphs with high extensibility. This method can support high concurrent interaction of clients and implement various tasks such as editing, querying, accessing and motion controlling. Some application experiments are performed to demonstrate its efficiency and soundness.展开更多
基金Supported by National Natural Science Foundation of China(61872024)National Key R&D Program of China under Grant(2018YFB2100603).
文摘Background In this study,we propose a novel 3D scene graph prediction approach for scene understanding from point clouds.Methods It can automatically organize the entities of a scene in a graph,where objects are nodes and their relationships are modeled as edges.More specifically,we employ the DGCNN to capture the features of objects and their relationships in the scene.A Graph Attention Network(GAT)is introduced to exploit latent features obtained from the initial estimation to further refine the object arrangement in the graph structure.A one loss function modified from cross entropy with a variable weight is proposed to solve the multi-category problem in the prediction of object and predicate.Results Experiments reveal that the proposed approach performs favorably against the state-of-the-art methods in terms of predicate classification and relationship prediction and achieves comparable performance on object classification prediction.Conclusions The 3D scene graph prediction approach can form an abstract description of the scene space from point clouds.
基金Supported by National Natural Science Foundation of China(Nos.61173080,61232014,61472010,61421062)National Key Technology Support Program of China(No.2013BAK03B07)
文摘Scene graph is a infrastructure of the virtual reality system to organize the virtual scene with abstraction, it can provide facility for the rendering engine and should be integrated effectively on demand into a real-time system, where a large quantities of scene objects and resources can be manipulated and managed with high flexibility and reliability. We present a new scheme of multiple scene graphs to accommodate the features of rendering engine and distributed systems. Based upon that, some other functions, e.g. block query, interactive editing, permission management, instance response, "redo" and "undo", are implemented to satisfy various requirements. At the same time, our design has compatibility to popular C/S architecture with good concurrent performance. Above all, it is convenient to be used for further development. The results of experiments including responding time demonstrate its good performance.
基金funded by(i)Natural Science Foundation China(NSFC)under Grant Nos.61402397,61263043,61562093 and 61663046(ii)Open Foundation of Key Laboratory in Software Engineering of Yunnan Province:No.2020SE304.(iii)Practical Innovation Project of Yunnan University,Project Nos.2021z34,2021y128 and 2021y129.
文摘Traffic scene captioning technology automatically generates one or more sentences to describe the content of traffic scenes by analyzing the content of the input traffic scene images,ensuring road safety while providing an important decision-making function for sustainable transportation.In order to provide a comprehensive and reasonable description of complex traffic scenes,a traffic scene semantic captioningmodel withmulti-stage feature enhancement is proposed in this paper.In general,the model follows an encoder-decoder structure.First,multilevel granularity visual features are used for feature enhancement during the encoding process,which enables the model to learn more detailed content in the traffic scene image.Second,the scene knowledge graph is applied to the decoding process,and the semantic features provided by the scene knowledge graph are used to enhance the features learned by the decoder again,so that themodel can learn the attributes of objects in the traffic scene and the relationships between objects to generate more reasonable captions.This paper reports extensive experiments on the challenging MS-COCO dataset,evaluated by five standard automatic evaluation metrics,and the results show that the proposed model has improved significantly in all metrics compared with the state-of-the-art methods,especially achieving a score of 129.0 on the CIDEr-D evaluation metric,which also indicates that the proposed model can effectively provide a more reasonable and comprehensive description of the traffic scene.
基金This work was supported by the National Natural Science Foundation of China(Nos.62173045 and 61673192)the Fundamental Research Funds for the Central Universities(No.2020XD-A04-2)the BUPT Excellent PhD Students Foundation(No.CX2021222).
文摘Scene graphs of point clouds help to understand object-level relationships in the 3D space.Most graph generation methods work on 2D structured data,which cannot be used for the 3D unstructured point cloud data.Existing point-cloud-based methods generate the scene graph with an additional graph structure that needs labor-intensive manual annotation.To address these problems,we explore a method to convert the point clouds into structured data and generate graphs without given structures.Specifically,we cluster points with similar augmented features into groups and establish their relationships,resulting in an initial structural representation of the point cloud.Besides,we propose a Dynamic Graph Generation Network(DGGN)to judge the semantic labels of targets of different granularity.It dynamically splits and merges point groups,resulting in a scene graph with high precision.Experiments show that our methods outperform other baseline methods.They output reliable graphs describing the object-level relationships without additional manual labeled data.
基金supported in part by National Natural Science Foundation of China(Nos.61721004,61976214,62076078 and 62176246).
文摘Image paragraph generation aims to generate a long description composed of multiple sentences,which is different from traditional image captioning containing only one sentence.Most of previous methods are dedicated to extracting rich features from image regions,and ignore modelling the visual relationships.In this paper,we propose a novel method to generate a paragraph by modelling visual relationships comprehensively.First,we parse an image into a scene graph,where each node represents a specific object and each edge denotes the relationship between two objects.Second,we enrich the object features by implicitly encoding visual relationships through a graph convolutional network(GCN).We further explore high-order relations between different relation features using another graph convolutional network.In addition,we obtain the linguistic features by projecting the predicted object labels and their relationships into a semantic embedding space.With these features,we present an attention-based topic generation network to select relevant features and produce a set of topic vectors,which are then utilized to generate multiple sentences.We evaluate the proposed method on the Stanford image-paragraph dataset which is currently the only available dataset for image paragraph generation,and our method achieves competitive performance in comparison with other state-of-the-art(SOTA)methods.
文摘对现实中的树木结构进行研究,在L系统的基础上对字符集进行扩展,设计了更加丰富的树木L文法描述,提出了比较符合树木生长过程的几何幂衰减规律,并结合图形渲染引擎OSG(open scene graph)实现了三维树木的树干部分的绘制。通过给树木的树干部分添加纹理、绘制树叶并给树叶添加阴影,使得绘制的树木更加真实自然。将绘制的树木作为一个节点加载到虚拟场景中,增强了虚拟场景的真实感。
文摘基于osg Earth开源工程是将地理信息系统和虚拟现实技术结合起来的一个重要应用。以基于osg Earth三维数字地球建设的研究为例,采用LOD与分页、动态调度等技术。研究了三维地形、影像、地物组织、调度及融合方式;设计了地球影像数据获取、坐标纹理校正、earth文件编写、矢量数据栅格化显示;分析了osg Earth中绘制线状矢量数据的模板体技术和该技术所隐藏的场景层次结构。在Visual Studio 2010平台下,使用Open Scene Graph语言进行编程,证明了该技术的可行性,实现了三维数字地球的建设。
文摘针对无人机飞行控制仿真环境以数据方式输出仿真结果不利于挖掘隐藏于数据中的本质特性的缺陷,借助虚拟现实技术的相关理论,设计实现基于无人机飞行控制仿真环境的三维视景子系统,实时动态再现无人机飞行运动全过程,为研究无人机飞行运动规律提供直观的事实根据。阐述了无人机飞行控制仿真环境三维视景子系统设计实现的目标,构造了系统结构并提出了三维视景系统的实现框架,对视景系统实现的主要技术进行了着重介绍,在视景系统的具体实现中对视景数据库设计、视景系统开发、视点切换、LynX Prim e和ACF配置文件、数据通信接口的具体实现进行了详细的论述。最后给出了系统的实施效果。
基金Supported by the National Basic Research Program of China (Grant No. 2004CB719403)the National High-Tech Research & Development Program of China (Grant Nos. 2006AA01Z334, 2007AA01Z318, 2009AA01Z324)+1 种基金the National Natural Science Foundation of China (GrantNos. 60573151,60703062,60833007)the Marine 908-03-01-10 Project
文摘An ultra-massive distributed virtual environment generally consists of ultra-massive terrain data and a large quantity of objects and their attribute data, such as 2D/3D geometric models, audio/video, images, vectors, characteristics, etc. In this paper, we propose a novel method for constructing distributed scene graphs with high extensibility. This method can support high concurrent interaction of clients and implement various tasks such as editing, querying, accessing and motion controlling. Some application experiments are performed to demonstrate its efficiency and soundness.