Binaural rendering is of great interest to virtual reality and immersive media. Although humans can naturally use their two ears to perceive the spatial information contained in sounds, it is a challenging task for ma...Binaural rendering is of great interest to virtual reality and immersive media. Although humans can naturally use their two ears to perceive the spatial information contained in sounds, it is a challenging task for machines to achieve binaural rendering since the description of a sound field often requires multiple channels and even the metadata of the sound sources. In addition, the perceived sound varies from person to person even in the same sound field. Previous methods generally rely on individual-dependent head-related transferred function(HRTF)datasets and optimization algorithms that act on HRTFs. In practical applications, there are two major drawbacks to existing methods. The first is a high personalization cost, as traditional methods achieve personalized needs by measuring HRTFs. The second is insufficient accuracy because the optimization goal of traditional methods is to retain another part of information that is more important in perception at the cost of discarding a part of the information. Therefore, it is desirable to develop novel techniques to achieve personalization and accuracy at a low cost. To this end, we focus on the binaural rendering of ambisonic and propose 1) channel-shared encoder and channel-compared attention integrated into neural networks and 2) a loss function quantifying interaural level differences to deal with spatial information. To verify the proposed method, we collect and release the first paired ambisonic-binaural dataset and introduce three metrics to evaluate the content information and spatial information accuracy of the end-to-end methods. Extensive experimental results on the collected dataset demonstrate the superior performance of the proposed method and the shortcomings of previous methods.展开更多
Three-dimensional surfaces are typically modeled as implicit surfaces.However,direct rendering of implicit surfaces is not simple,especially when such surfaces contain finely detailed shapes.One approach is ray-castin...Three-dimensional surfaces are typically modeled as implicit surfaces.However,direct rendering of implicit surfaces is not simple,especially when such surfaces contain finely detailed shapes.One approach is ray-casting,where the field of the implicit surface is assumed to be piecewise polynomials defined on the grid of a rectangular domain.A critical issue for direct rendering based on ray-casting is the computational cost of finding intersections between surfaces and rays.In particular,ray-casting requires many function evaluations along each ray,severely slowing the rendering speed.In this paper,a method is proposed to achieve direct rendering of polynomial-based implicit surfaces in real-time by strategically narrowing the search range and designing the shader to exploit the structure of piecewise polynomials.In experiments,the proposed method achieved a high framerate performance for different test cases,with a speed-up factor ranging from 1.1 to 218.2.In addition,the proposed method demonstrated better efficiency with high cell resolution.In terms of memory consumption,the proposed method saved between 90.94%and 99.64%in different test cases.Generally,the proposed method became more memoryefficient as the cell resolution increased.展开更多
With the development of virtual reality (VR) technology, more and more industries are beginning to integrate with VR technology. In response to the problem of not being able to directly render the lighting effect of C...With the development of virtual reality (VR) technology, more and more industries are beginning to integrate with VR technology. In response to the problem of not being able to directly render the lighting effect of Caideng in digital Caideng scenes, this article analyzes the lighting model. It combines it with the lighting effect of Caideng scenes to design an optimized lighting model algorithm that fuses the bidirectional transmission distribution function (BTDF) model. This algorithm can efficiently render the lighting effect of Caideng models in a virtual environment. And using image optimization processing methods, the immersive experience effect on the VR is enhanced. Finally, a Caideng roaming interactive system was designed based on this method. The results show that the frame rate of the system is stable during operation, maintained above 60 fps, and has a good immersive experience.展开更多
基金supported in part by the National Natural Science Foundation of China (62176059, 62101136)。
文摘Binaural rendering is of great interest to virtual reality and immersive media. Although humans can naturally use their two ears to perceive the spatial information contained in sounds, it is a challenging task for machines to achieve binaural rendering since the description of a sound field often requires multiple channels and even the metadata of the sound sources. In addition, the perceived sound varies from person to person even in the same sound field. Previous methods generally rely on individual-dependent head-related transferred function(HRTF)datasets and optimization algorithms that act on HRTFs. In practical applications, there are two major drawbacks to existing methods. The first is a high personalization cost, as traditional methods achieve personalized needs by measuring HRTFs. The second is insufficient accuracy because the optimization goal of traditional methods is to retain another part of information that is more important in perception at the cost of discarding a part of the information. Therefore, it is desirable to develop novel techniques to achieve personalization and accuracy at a low cost. To this end, we focus on the binaural rendering of ambisonic and propose 1) channel-shared encoder and channel-compared attention integrated into neural networks and 2) a loss function quantifying interaural level differences to deal with spatial information. To verify the proposed method, we collect and release the first paired ambisonic-binaural dataset and introduce three metrics to evaluate the content information and spatial information accuracy of the end-to-end methods. Extensive experimental results on the collected dataset demonstrate the superior performance of the proposed method and the shortcomings of previous methods.
基金supported by JSPS KAKENHI Grant Number 21K11928。
文摘Three-dimensional surfaces are typically modeled as implicit surfaces.However,direct rendering of implicit surfaces is not simple,especially when such surfaces contain finely detailed shapes.One approach is ray-casting,where the field of the implicit surface is assumed to be piecewise polynomials defined on the grid of a rectangular domain.A critical issue for direct rendering based on ray-casting is the computational cost of finding intersections between surfaces and rays.In particular,ray-casting requires many function evaluations along each ray,severely slowing the rendering speed.In this paper,a method is proposed to achieve direct rendering of polynomial-based implicit surfaces in real-time by strategically narrowing the search range and designing the shader to exploit the structure of piecewise polynomials.In experiments,the proposed method achieved a high framerate performance for different test cases,with a speed-up factor ranging from 1.1 to 218.2.In addition,the proposed method demonstrated better efficiency with high cell resolution.In terms of memory consumption,the proposed method saved between 90.94%and 99.64%in different test cases.Generally,the proposed method became more memoryefficient as the cell resolution increased.
文摘With the development of virtual reality (VR) technology, more and more industries are beginning to integrate with VR technology. In response to the problem of not being able to directly render the lighting effect of Caideng in digital Caideng scenes, this article analyzes the lighting model. It combines it with the lighting effect of Caideng scenes to design an optimized lighting model algorithm that fuses the bidirectional transmission distribution function (BTDF) model. This algorithm can efficiently render the lighting effect of Caideng models in a virtual environment. And using image optimization processing methods, the immersive experience effect on the VR is enhanced. Finally, a Caideng roaming interactive system was designed based on this method. The results show that the frame rate of the system is stable during operation, maintained above 60 fps, and has a good immersive experience.