End-to-End Paired Ambisonic-Binaural Audio Rendering

Binaural rendering is of great interest to virtual reality and immersive media. Although humans can naturally use their two ears to perceive the spatial information contained in sounds, it is a challenging task for machines to achieve binaural rendering since the description of a sound field often requires multiple channels and even the metadata of the sound sources. In addition, the perceived sound varies from person to person even in the same sound field. Previous methods generally rely on individual-dependent head-related transferred function(HRTF)datasets and optimization algorithms that act on HRTFs. In practical applications, there are two major drawbacks to existing methods. The first is a high personalization cost, as traditional methods achieve personalized needs by measuring HRTFs. The second is insufficient accuracy because the optimization goal of traditional methods is to retain another part of information that is more important in perception at the cost of discarding a part of the information. Therefore, it is desirable to develop novel techniques to achieve personalization and accuracy at a low cost. To this end, we focus on the binaural rendering of ambisonic and propose 1) channel-shared encoder and channel-compared attention integrated into neural networks and 2) a loss function quantifying interaural level differences to deal with spatial information. To verify the proposed method, we collect and release the first paired ambisonic-binaural dataset and introduce three metrics to evaluate the content information and spatial information accuracy of the end-to-end methods. Extensive experimental results on the collected dataset demonstrate the superior performance of the proposed method and the shortcomings of previous methods.

Research and Application of Caideng Model Rendering Technology for Virtual Reality

With the development of virtual reality (VR) technology, more and more industries are beginning to integrate with VR technology. In response to the problem of not being able to directly render the lighting effect of Caideng in digital Caideng scenes, this article analyzes the lighting model. It combines it with the lighting effect of Caideng scenes to design an optimized lighting model algorithm that fuses the bidirectional transmission distribution function (BTDF) model. This algorithm can efficiently render the lighting effect of Caideng models in a virtual environment. And using image optimization processing methods, the immersive experience effect on the VR is enhanced. Finally, a Caideng roaming interactive system was designed based on this method. The results show that the frame rate of the system is stable during operation, maintained above 60 fps, and has a good immersive experience.

A survey of real-time rendering on Web3D application

Background In recent years, with the rapid development of mobile Internet and Web3D technologies, a large number of web-based online 3D visualization applications have emerged. Web3D applications, including Web3D online tourism, Web3D online architecture, Web3D online education environment, Web3D online medical care, and Web3D online shopping are examples of these applications that leverage 3D rendering on the web. These applications have pushed the boundaries of traditional web applications that use text, sound, image, video, and 2D animation as their main communication media, and resorted to 3D virtual scenes as the main interaction object, enabling a user experience that delivers a strong sense of immersion. This paper approached the emerging Web3D applications that generate stronger impacts on people's lives through “real-time rendering technology”, which is the core technology of Web3D. This paper discusses all the major 3D graphics APIs of Web3D and the well-known Web3D engines at home and abroad and classify the real-time rendering frameworks of Web3D applications into different categories. Results Finally, this study analyzed the specific demand posed by different fields to Web3D applications by referring to the representative Web3D applications in each particular field. Conclusions Our survey results show that Web3D applications based on real-time rendering have in-depth sectors of society and even family, which is a trend that has influence on every line of industry.

An Improved HVQ Algorithm for Compression and Rendering of Space Environment Volume Data with Multi-correlated Variables

Volume visualization can not only illustrate overall distribution but also inner structure and it is an important approach for space environment research.Space environment simulation can produce several correlated variables at the same time.However,existing compressed volume rendering methods only consider reducing the redundant information in a single volume of a specific variable,not dealing with the redundant information among these variables.For space environment volume data with multi-correlated variables,based on the HVQ-1d method we propose a further improved HVQ method by compositing variable-specific levels to reduce the redundant information among these variables.The volume data associated with each variable is divided into disjoint blocks of size 43 initially.The blocks are represented as two levels,a mean level and a detail level.The variable-specific mean levels and detail levels are combined respectively to form a larger global mean level and a larger global detail level.To both global levels,a splitting based on a principal component analysis is applied to compute initial codebooks.Then,LBG algorithm is conducted for codebook refinement and quantization.We further take advantage of progressive rendering based on GPU for real-time interactive visualization.Our method has been tested along with HVQ and HVQ-1d on high-energy proton flux volume data,including>5,>10,>30 and>50 MeV integrated proton flux.The results of our experiments prove that the method proposed in this paper pays the least cost of quality at compression,achieves a higher decompression and rendering speed compared with HVQ and provides satisficed fidelity while ensuring interactive rendering speed.

Perceptual Optimization for Point-Based Point Cloud Rendering

Point-based rendering is a common method widely used in point cloud rendering.It realizes rendering by turning the points into the base geometry.The critical step in point-based rendering is to set an appropriate rendering radius for the base geometry,usually calculated using the average Euclidean distance of the N nearest neighboring points to the rendered point.This method effectively reduces the appearance of empty spaces between points in rendering.However,it also causes the problem that the rendering radius of outlier points far away from the central region of the point cloud sequence could be large,which impacts the perceptual quality.To solve the above problem,we propose an algorithm for point-based point cloud rendering through outlier detection to optimize the perceptual quality of rendering.The algorithm determines whether the detected points are outliers using a combination of local and global geometric features.For the detected outliers,the minimum radius is used for rendering.We examine the performance of the proposed method in terms of both objective quality and perceptual quality.The experimental results show that the peak signal-to-noise ratio(PSNR)of the point cloud sequences is improved under all geometric quantization,and the PSNR improvement ratio is more evident in dense point clouds.Specifically,the PSNR of the point cloud sequences is improved by 3.6%on average compared with the original algorithm.The proposed method significantly improves the perceptual quality of the rendered point clouds and the results of ablation studies prove the feasibility and effectiveness of the proposed method.

Web-based mixed reality video fusion with remote rendering

Background Mixed reality(MR)video fusion systems merge video imagery with 3D scenes to make the scene more realistic and help users understand the video content and temporal–spatial correlation between them,reducing the user′s cognitive load.MR video fusion are used in various applications;however,video fusion systems require powerful client machines because video streaming delivery,stitching,and rendering are computationally intensive.Moreover,huge bandwidth usage is another critical factor that affects the scalability of video-fusion systems.Methods Our framework proposes a fusion method for dynamically projecting video images into 3D models as textures.Results Several experiments on different metrics demonstrate the effectiveness of the proposed framework.Conclusions The framework proposed in this study can overcome client limitations by utilizing remote rendering.Furthermore,the framework we built is based on browsers.Therefore,the user can test the MR video fusion system with a laptop or tablet without installing any additional plug-ins or application programs.

Image Color Rendering Based on Hinge-Cross-Entropy GAN in Internet of Medical Things

Computer-aided diagnosis based on image color rendering promotes medical image analysis and doctor-patient communication by highlighting important information of medical diagnosis.To overcome the limitations of the color rendering method based on deep learning,such as poor model stability,poor rendering quality,fuzzy boundaries and crossed color boundaries,we propose a novel hinge-cross-entropy generative adversarial network(HCEGAN).The self-attention mechanism was added and improved to focus on the important information of the image.And the hinge-cross-entropy loss function was used to stabilize the training process of GAN models.In this study,we implement the HCEGAN model for image color rendering based on DIV2K and COCO datasets,and evaluate the results using SSIM and PSNR.The experimental results show that the proposed HCEGAN automatically re-renders images,significantly improves the quality of color rendering and greatly improves the stability of prior GAN models.

Fabrication of High Color Rendering Index White Light Emitting Diodes from Gold Nanoclusters

We demonstrated gold nanoclusters as color tunable emissive light converters for the application of white light emitting diodes (WLEDs). A blue LED providing 460 nm to excite gold nanoclusters mixed with UV curable material generates broad bandwidth emission at the visible range. Increasing the amount of gold nanoclusters, the correlated color temperature of WLEDs tuned from cold white to warm white, and also results in the variation of color rendering index (CRI). The highest CRI in the experiment is 92.

面向游戏与动漫的黑白漫画风格非真实感实时渲染方法

针对一般黑白漫画风格的非真实感渲染方法资源开销高和实时性差的问题,提出一种面向游戏与动漫的黑白漫画风格非真实感实时渲染方法。设计特化光照模型,通过分析光照模型效果,设计主体突出的环境光项和灰度分级的漫反射项,合并光照模型各项,实现基于特化光照模型的场景预处理;提出屏幕空间三相边缘检测方法,将预处理得到的场景光照结果依次进行深度边缘检测、法线边缘检测、颜色边缘检测,合并边缘检测结果;对场景光照结果进行屏幕空间分区域渲染,依据光照结果灰度值大小,分别采用网点着色和线条着色方式处理,并采用插值方法处理着色交界;合并结果,依据光照模型结果灰度值设计分段函数,合并屏幕空间三相边缘检测结果与屏幕空间分区域渲染结果,得到最终渲染结果。实验表明,该方法能够以较低的资源开销和较快的渲染速度实现黑白漫画风格非真实感渲染,在有限的资源下提高渲染的效果,具有显著的黑白漫画风格。

孔子“以直报怨”思想研究——兼论孔子是否反对“以德报怨”

据《论语·宪问》,孔子主张“以直报怨”,而对“以德报怨”则并未明确表示赞成或反对。通过考察《论语》中关于“直”、“怨”的各种论述,结合历代学者的相关观点,可以得出:“以直报怨”中的“直”,应该包含诚实、无私、符合通权达变原则的正直、等值、合乎礼的直率等丰富的含义;“以德报怨”的情况比较复杂,其中有的体现了人们不计前嫌、公而忘私的高尚品德,有的则反映了一些人的虚伪和充满私心的算计,因此,对于“以德报怨”,我们不能简单地予以肯定或否定;“以直报怨”既包含“以德报怨”中值得肯定或提倡的内容,又包含“以德报怨”中所没有的对怨恨对象采取相应报复措施的内容,因此是一种更为全面、合理的报怨方式。

冬奥气象图形产品可视化模型研究

高质量图形产品是冬奥气象服务信息的重要载体。基于冬奥气象数据高时空分辨率特征,研究了由多维气象信息向制图要素转换的冬奥气象图形产品可视化概念模型;设计了由基于数据引擎的结构映射、赛事服务的空间映射、组件应用的元素映射和派生思想的属性映射组成的冬奥气象图形产品可视化过程模型。针对差异化、准实时气象服务制图需求,设计了基于DVDL可视化描述语言的制图表达框架,对可视化模型进行业务组织,研发了冬奥气象图形产品加工系统,提供50余种基础产品样式和所见即所得的制图编辑功能,支持任意位置点长时间序列气象产品、山地赛场三维气象产品和多赛区综合服务气象产品的制图渲染。成果在2022年北京冬奥会气象服务中应用,生产了200余种不用服务主题气象图形产品,后续也可为其他冬季赛事和重大活动气象服务保障提供精细化产品制作解决方案。

一种带高光处理的无缝纹理映射方法

纹理映射作为三维重建的重要步骤,直接关系到生成模型的视觉效果。传统的纹理映射方法通常使用简单的混合方法来获取表面纹理。通过将模型表面投影到每个纹理图像上来获取相应的纹理区域,然后将它们混合在一起以获取表面纹理。然而,由于相机位姿的不准确性,纹理映射结果存在明显的模糊和鬼影等问题。此外,在高光环境下获取的纹理图像很容易包含高光区域,最终导致纹理颜色的丢失,降低纹理的真实性。为了解决这些问题,提出了一种无缝纹理映射方法,可以有效消除高光反射。通过衡量纹理图像的质量,为每个模型表面选择最佳的纹理图像,并利用色度一致性约束优化的相机位姿以消除明显的纹理错位。针对高光问题,提出了一个高光处理模块,使用多视图图像信息并基于双色反射模型定位和处理高光纹理。最后,对纹理图进行纹理颜色一致性调整,以处理纹理之间的颜色差异。实验结果表明,与现有方法相比,提出的算法可以有效消除高光反射的影响,从而获得更好的纹理映射结果。

基于主客观数据的汽车人机交互界面表面触觉反馈方法

针对汽车用户界面虚拟按钮缺乏触觉反馈或触觉反馈感受单一的问题,采集3种真实物理按钮(切换开关、自锁按钮、复位按钮)数据并提取按钮特征,基于手指按压力大小分段实现了不同的振动反馈。提出一种重现按钮顺应性的触觉渲染方法,根据用户主观感受,验证了其相比于顺应性错觉方法在顺应性、用户愉悦性等方面的提升。进行触觉反馈匹配实验,实验者根据触觉感受将3种虚拟按钮与物理按钮类型进行匹配,准确率高达94.33%,验证了提出的虚拟按钮振动触觉反馈对不同物理按钮独特感知特性恢复的有效性,用户可以在无视觉需求的情况下辨别操纵的虚拟按钮类型和状态。

注意力机制与神经渲染的多视图三维重建算法

针对多视图立体网络在弱纹理或非朗伯曲面等挑战性区域重建效果差的问题,首先提出一个基于3个并行扩展卷积和注意力机制的多尺度特征提取模块,在增加感受野的同时捕获特征之间的依赖关系以获取全局上下文信息,从而提升多视图立体网络在挑战性区域特征的表征能力以进行鲁棒的特征匹配。其次在代价体正则化3D CNN部分引入注意力机制,使网络注意于代价体中的重要区域以进行平滑处理。另外建立一个神经渲染网络,该网络利用渲染参考损失精确地解析辐射场景表达的几何外观信息,并引入深度一致性损失保持多视图立体网络与神经渲染网络之间的几何一致性,有效地缓解有噪声代价体对多视图立体网络的不利影响。该算法在室内DTU数据集中测试,点云重建的完整性和整体性指标分别为0.289和0.326,与基准方法CasMVSNet相比,分别提升24.9%和8.2%,即使在挑战性区域也得到高质量的重建效果;在室外Tanks and Temples中级数据集中,点云重建的平均F-score为60.31,与方法UCS-Net相比提升9.9%,体现出较强的泛化能力。

结合上下文特征融合的虚拟视点图像空洞填充

由于参考纹理视图的前景遮挡和不同视点间的视角差异,基于深度图的虚拟视点合成会产生大量空洞,先前的空洞填充方法耗时较长且填充区域与合成图像缺乏纹理一致性。该文首先对深度图进行预处理来减少空洞填充时的前景渗透;然后,针对经3D-warping后输出合成图像中的空洞,设计了一种基于生成对抗网络(GAN)架构的图像生成网络来填充空洞。该网络模型由2级子网络构成,第1级网络生成空洞区域的纹理结构信息,第2级网络采用了一种结合上下文特征融合的注意力模块来提升空洞填充质量。提出的网络模型能有效解决当虚拟视点图像中的前景对象存在快速运动时,空洞填充区易产生伪影的问题。在多视点深度序列上的实验结果表明,提出方法在主客观质量上均优于已有的虚拟视点图像空洞填充方法。

基于计算机视觉的舰船模型三维重建方法

为保证舰船模型三维重建的完整性以及细节程度,提出基于计算机视觉的舰船模型三维重建方法。基于双目立体视觉计算全局舰船目标点云坐标,生成舰船全局的三维点云数据;多视角点云配准方法对该数据进行配准后,输入多尺度特征递归卷积的稠密点云重建网络模型中,通过该模型生成舰船模型三维深度图,在此基础上,利用运动结构法完成舰船深度图中的三维曲线重建,对舰船模型进行颜色渲染,输出舰船模型三维重建结果。测试结果显示,该方法能够完成不同图像之间对应点的可靠匹配,确定各点的坐标位置;三维重建后模型的空间偏差均在0.015以下;能够较好地完成舰船结构的重建,重建后舰船模型的完整性较好,清晰呈现舰船的结构细节。

基于虚拟现实的虚拟仿真建模及渲染技术

虚拟现实技术能够创造出逼真的虚拟环境,为能够更加快速地建立模型,该文结合相关工具和Unity3D平台开发仿真建模及渲染技术。通过工具收集真实地形的数据,然后进行相关的数据处理并生成数据集合。通过使用Unity3D中地形工具进行数据读取和快速制作三维场景模型,为提高模型的渲染逼真度使用高清渲染管线对场景模型进行光照等渲染处理。通过对该技术的研究,该技术能够使三维场景的逼真度提高及渲染程度非常好,模型细节也能观察,同时该技术也能被运用到毁伤评估仿真、虚拟训练等方面。

虚实共生视域下的职业教育内容呈现模式研究

当今职业院校的学生成长于信息技术高速发展的时代,是同时拥有虚拟空间和现实存在两大生活空间的年轻人。对于他们,职业教育进行虚实共生的内容呈现是可行并必要的。从虚实共生和职业教育的关系出发,分析了以往常见的内容呈现方式的不足,比较了虚实环境各自的优点和缺点,提炼了虚实融合共生的意义所在,并在此基础上分析了几种虚实共生下的有效内容呈现模式,并提出多元化的模式结合呈现模式实现全方位教育,展望了虚实共生下的职业教育内容呈现场景,为虚实融合的教育内容与环境建设提供了思路。

神经辐射场多视图合成技术综述

如何从图像中渲染出较为真实的虚拟场景一直是计算机图形学与计算机视觉领域的研究目标之一。神经辐射场是一种基于深度神经网络的新兴方法,它通过学习场景中每个点的辐射场来实现较为真实的渲染效果。通过神经辐射场不仅可以生成逼真的图像,而且可以生成具有真实感的三维场景,在虚拟现实、增强现实和计算机游戏等领域有着广泛的应用前景。然而,其基础模型存在训练效率低、泛化能力差、可解释性不足、易受光照和材质变化影响以及无法处理动态场景等问题,在某些情况下无法获得最佳的渲染结果。大量基于此研究的工作陆续展开,且在效率和精度等方面都取得了出色的成果。为了跟踪该领域最新研究成果,对近年来神经辐射场领域的关键算法进行回顾和综述。首先介绍了神经辐射场的产生背景及原理,对后续关键改进模型进行分类探讨。主要涵盖以下几个方面:对神经辐射场基本模型参数的优化,在渲染速度与推理能力方面的提升,对空间表达和光照能力的改善,针对静态场景相机位姿估计与稀疏视图合成方法的改进,以及在动态场景建模领域的发展。对各种模型的速度与性能进行分类对比与分析,并简要介绍了该领域主要模型评估指标与公开数据集。最后对神经辐射场未来发展趋势进行展望。

基于神经渲染的数字孪生资产快速场景几何建模与检索方法

虚实融合是数字孪生技术的典型特征,数字孪生的虚拟场景大多通过几何建模技术来实现。为解决场景几何建模的自动化程度较低。过于依赖人工,导致成本高昂且效率低下的问题,提出一种数字孪生几何场景构建方法。引入神经渲染技术采集物理实体的点云数据,然后设计一种基于深度学习的点云模型与3DCAD模型语义映射的方法,用于基于数字孪生几何模型资产库的3DCAD模型检索。最后建立训练数据集并构建数字孪生几何模型资产库进行实验验证。通过对比实验和退役电池拆解案例验证:与其他的数字孪生场景几何建模方法相比,该方法具有更低的成本和更高的效率。