神经渲染及其硬件加速综述

Survey on Neural Rendering and Its Hardware Acceleration

神经渲染是一种基于深度学习的新兴图像和视频生成方法,它将深度学习模型和计算机图形学的物理模型相结合,从而获得可控和逼真的场景表示和渲染,实现对诸如光照、相机参数、姿态等场景属性的控制.一方面,神经渲染既可以充分利用深度学习的优势,实现传统前向渲染流程的加速,也可以为影像的逆向渲染和3维场景重建等特定任务提供新的解决方案;另一方面设计适应神经渲染流水线的创新硬件结构,突破现有图形处理器的并行计算和功耗瓶颈,有望为虚拟现实和增强现实、影视创作和数字娱乐、人工智能和元宇宙等未来重点领域提供重要支撑.综述了神经渲染的技术内涵、主要挑战和研究进展,在此基础上分析了神经渲染流水线对硬件加速支持的共性需求和当前主流神经渲染硬件加速结构的特点,进而讨论了神经渲染处理器架构的设计挑战,最后展望了其未来的发展趋势.

Neural rendering is a new image and video generation method based on deep learning.It combines the deep learning model with the physical knowledge of computer graphics to obtain a controllable and realistic scene model,and realize the control of scene attributes such as lighting,camera parameters,posture and so on.On the one hand,neural rendering can not only make full use of the advantages of deep learning to accelerate the traditional forward rendering process,but also provide new solutions for specific tasks such as inverse rendering and 3D reconstruction.On the other hand,the design of innovative hardware structures that adapt to the neural rendering pipeline breaks through the parallel computing and power consumption bottleneck of existing graphics processors,which is expected to provide important support for future key areas such as virtual and augmented reality,film and television creation and digital entertainment,artificial intelligence and metaverse.In this paper,we review the technical connotation,main challenges,and research progress of neural rendering.On this basis,we analyze the common requirements of neural rendering pipeline for hardware acceleration and the characteristics of the current hardware acceleration architecture,and then discuss the design challenges of neural rendering processor architecture.Finally,the future development trend of neural rendering processor architecture is prospected.