神经辐射场应用于大规模实景三维场景可视化研究进展

Progress in neural radiance field and its application in large-scale real-scene 3D visualization

地理实景三维场景是重要的国家数字基础设施,其将地理信息从传统二维平面扩展到信息更丰富更全面的三维空间,数据以显式三维模型的形式存储表达。然而,经典的显式三维模型具有数据量大、可视化效果粗糙等问题,在一定程度上限制了实景三维模型的实际应用。神经辐射场NeRF(Neural Radiance Field)是一种基于神经隐式立体表达(Neural Implicit Volume Representing)进行可微渲染(Differentiable Rendering)以实现高质量视图合成的新方法,由Mildenhall等(2020)首次提出,以其逼真的视图合成效果与新颖的实现方式成为计算机视觉领域的热点研究方向。自NeRF提出以来,国内外爆发式涌现出大量有关神经辐射场的研究文献,主要聚集于可视化效果的生成方法研究,兼有少量将其用于大规模实景三维场景可视化研究探索。本文回顾了神经辐射场提出的背景,概述了神经辐射场及其在大规模实景三维可视化方面的研究进展,分析了目前利用神经辐射场进行大规模实景三维场景可视化研究中被关注的无边界场景、锯齿效果、瞬态遮挡、光度一致性、场景重照明与可见性场等问题,指出了目前研究在多源数据融合、视觉效果优化、虚拟环境感知等方面面临的挑战,对未来值得进一步深入探索的方向进行了展望。

Geographical real-scene 3D scenes are an important national digital infrastructure,which extends geographic information from 2D to 3D.Real-scene 3D data are stored and expressed in the form of an explicit 3D model,which has the problems of large amount of data and rough visualization effect.Neural Radiance Field(NeRF),realizing differentiable rendering based on neural implicit volume representation,is an innovative approach of high-quality view synthesis.First proposed by Mildenhall et al.(2020),NeRF has become one of the hottest research direction in the field of computer vision due to its realistic view synthesis effect.A large amount of literature about NeRF have been published since NeRF was proposed,and the application of NeRF in large-scale real-scene 3D visualization has begun to attract the attention of some published papers.View synthesis,which uses sparse 2D images to generate realistic new views at any viewpoint in 3D space without the reconstruction of 3D models,is a novel way to realize the representation of 3D scenes.The development of view synthesis technology has gone through several stages:image mosaicking,3D model reprojection,view interpolation,and volume representing technology.NeRF,as an innovative approach of view synthesis,samples 5D coordinates(location and viewing direction)along camera rays,feeds those locations into a multilayer perceptron network to produce color and volume density,and uses volume rendering techniques to composite these values into a new image.NeRF not only produces remarkably higher-quality rendering than prior volumetric approaches but also requires just a fraction of the storage cost of other sampled volumetric representations.However,it faces problems such as requirements for high quality of source data,failure to support dynamic objects,low efficiency in processing,and single type of render target.Moreover,NeRF-related research are mostly conducted based on laboratory environment or standardized data at present.Due to these drawbacks,many obstacles need to be overcome before applying NeRF to large-scale real-scene 3D visualization.This paper reviews the workaround of unbounded scene,aliasing,luminosity consistency,scene relighting,and visibility field in the Block-NeRF algorithm,a variant of NeRF that can represent large-scale environments(Tancik et al.2022).The Block-NeRF algorithm splits the environment into a set of Block-NeRFs that can be independently trained in parallel and composited during inference,and it selects relevant Block-NeRFs for rendering,which are then composited smoothly when traversing the scene.To aid with this compositing,BlockNeRF optimizes the appearance codes to match lighting conditions.It trains individual Block-NeRFs using techniques such as appearance embeddings,learned pose refinement,exposure input,transient objects,and visibility prediction.Nowadays,the studies of applying NeRF to large-scale real-scene 3D visualization are being conducted extensively,which has attracted much attention and plays a pioneering,leading role in further research.The results of these studies achieve the most basic 3D visualization large-scale real-scene effect,but they are limited by some conditions and their universality needs to be strongly improved.Because NeRF is still far from practical applications in producing large-scale real-scene 3D visualization,any slight progressive exploration is likely to become a continuous research hotspot.This paper identifies the challenges of NeRF research,including multisource data fusion,visual effect optimization,and virtual environment perception,which need more research.