Background Free-viewpoint video(FVV)is processed video content in which viewers can freely select the viewing position and angle.FVV delivers an improved visual experience and can also help synthesize special effects ...Background Free-viewpoint video(FVV)is processed video content in which viewers can freely select the viewing position and angle.FVV delivers an improved visual experience and can also help synthesize special effects and virtual reality content.In this paper,a complete FVV system is proposed to interactively control the viewpoints of video relay programs through multimedia terminals such as computers and tablets.Methods The hardware of the FVV generation system is a set of synchronously controlled cameras,and the software generates videos in novel viewpoints from the captured video using view interpolation.The interactive interface is designed to visualize the generated video in novel viewpoints and enable the viewpoint to be changed interactively.Results Experiments show that our system can synthesize plausible videos in intermediate viewpoints with a view range of up to 180°.展开更多
In recent years, many image-based rendering techniques have advanced from static to dynamic scenes and thus become video-based rendering （VBR） methods. But actually, only a few of them can render new views on-line. ...In recent years, many image-based rendering techniques have advanced from static to dynamic scenes and thus become video-based rendering （VBR） methods. But actually, only a few of them can render new views on-line. We present a new VBR system that creates new views of a live dynamic scene. This system provides high quality images and does not require any background subtraction. Our method follows a plane-sweep approach and reaches real-time rendering using consumer graphic hardware, graphics processing unit （GPU）. Only one computer is used for both acquisition and rendering. The video stream acquisition is performed by at least 3 webcams. We propose an additional video stream management that extends the number of webcams to 10 or more. These considerations make our system low-cost and hence accessible for everyone. We also present an adaptation of our plane-sweep method to create simultaneously multiple views of the scene in real-time. Our system is especially designed for stereovision using autostereoscopic displays. The new views are computed from 4 webcams connected to a computer and are compressed in order to be transfered to a mobile phone. Using GPU programming, our method provides up to 16 images of the scene in real-time. The use of both GPU and CPU makes this method work on only one consumer grade computer.展开更多
A semi-reference image quality assessment metric based on similarity measurement for synthesized virtual viewpoint image （VVI） in free-viewpoint television system （FFV） is proposed in this paper. The key point of ...A semi-reference image quality assessment metric based on similarity measurement for synthesized virtual viewpoint image （VVI） in free-viewpoint television system （FFV） is proposed in this paper. The key point of the proposed metric is taking resemblant information between VVI and its neighbor view images for quality assessment to make our metric to be extended to multi-semi-reference image quality assessment easily. The proposed metric first extracts impact factors from image features, then combines an image synthesis technique and similarity functions, in which, disparity information are taken into account for registering the resemblant regions. Experiments are divided into three phases. Phase I is to verify the validation of the proposed metric by taking impaired images and original reference into account. The experimental results show the agreement between evaluation scores and bio-characteristic of human visual system. Phase II shows the accordance with Phase I by taking neighbor view as reference. The proposed metric can be taken as a full reference one to evaluate the image quality even though the original reference is absent. Phase III is then performed to evaluate the quality of WI. Evaluation scores in the experimental results are able to evaluate the quality of VVI.展开更多
Free-viewpoint video allows the user to view objects from any virtual perspective,creating an immersive visual experience.This technology enhances the interactivity and freedom of multimedia performances.However,many ...Free-viewpoint video allows the user to view objects from any virtual perspective,creating an immersive visual experience.This technology enhances the interactivity and freedom of multimedia performances.However,many free-viewpoint video synthesis methods hardly satisfy the requirement to work in real time with high precision,particularly for sports fields having large areas and numerous moving objects.To address these issues,we propose a freeviewpoint video synthesis method based on distance field acceleration.The central idea is to fuse multiview distance field information and use it to adjust the search step size adaptively.Adaptive step size search is used in two ways:for fast estimation of multiobject three-dimensional surfaces,and synthetic view rendering based on global occlusion judgement.We have implemented our ideas using parallel computing for interactive display,using CUDA and OpenGL frameworks,and have used real-world and simulated experimental datasets for evaluation.The results show that the proposed method can render free-viewpoint videos with multiple objects on large sports fields at 25 fps.Furthermore,the visual quality of our synthetic novel viewpoint images exceeds that of state-of-the-art neural-rendering-based methods.展开更多
The trend in video viewing has been evolving beyond simply providing a multi-view option.Recently,a function that allows selection and viewing of a clip from a multi-view service that captures a specific range or obje...The trend in video viewing has been evolving beyond simply providing a multi-view option.Recently,a function that allows selection and viewing of a clip from a multi-view service that captures a specific range or object has been added.In particular,the free-view service is an extended concept of multi-view and provides a freer viewpoint.However,since numerous videos and additional data are required for its construction,all of the clips constituting the content cannot be simultaneously provided.Only certain clips are selected and provided to the user.If the video is not the preferred video,change request is made,and a delay occurs during retransmission from the server.Delays due to frequent re-requests degrade the overall quality of service.For free-view services,selectively transmitting the video according to the user’s desired viewpoint and region of interest within the limited network of available videos is important.In this study,we propose a method of screening and providing the correct video based on objects in the contents.Based on the method of recognizing the object in each clip,we designed a method of setting its priority based on information about the object’s location for each viewpoint.During the transmission and receiving process using this information,the selected video can be rapidly recognized and changed.Herein,we present a service system configuration method and propose video selection examples for free-view services.展开更多
Depth maps are used for synthesis virtual view in free-viewpoint television （FTV） systems. When depth maps are derived using existing depth estimation methods, the depth distortions will cause undesirable artifacts ...Depth maps are used for synthesis virtual view in free-viewpoint television （FTV） systems. When depth maps are derived using existing depth estimation methods, the depth distortions will cause undesirable artifacts in the synthesized views. To solve this problem, a 3D video quality model base depth maps （D-3DV） for virtual view synthesis and depth map coding in the FTV applications is proposed. First, the relationships between distortions in coded depth map and rendered view are derived. Then, a precisely 3DV quality model based depth characteristics is develop for the synthesized virtual views. Finally, based on D-3DV model, a multilateral filtering is applied as a pre-processed filter to reduce rendering artifacts. The experimental results evaluated by objective and subjective methods indicate that the proposed D-3DV model can reduce bit-rate of depth coding and achieve better rendering quality.展开更多
基金Supported by USDA 2018-67021-27416NSFJS BK20190292.
文摘Background Free-viewpoint video(FVV)is processed video content in which viewers can freely select the viewing position and angle.FVV delivers an improved visual experience and can also help synthesize special effects and virtual reality content.In this paper,a complete FVV system is proposed to interactively control the viewpoints of video relay programs through multimedia terminals such as computers and tablets.Methods The hardware of the FVV generation system is a set of synchronously controlled cameras,and the software generates videos in novel viewpoints from the captured video using view interpolation.The interactive interface is designed to visualize the generated video in novel viewpoints and enable the viewpoint to be changed interactively.Results Experiments show that our system can synthesize plausible videos in intermediate viewpoints with a view range of up to 180°.
基金This work was supported by Foundation of Technology Supporting the Creation of Digital Media Contents project (CREST, JST), Japan
文摘In recent years, many image-based rendering techniques have advanced from static to dynamic scenes and thus become video-based rendering （VBR） methods. But actually, only a few of them can render new views on-line. We present a new VBR system that creates new views of a live dynamic scene. This system provides high quality images and does not require any background subtraction. Our method follows a plane-sweep approach and reaches real-time rendering using consumer graphic hardware, graphics processing unit （GPU）. Only one computer is used for both acquisition and rendering. The video stream acquisition is performed by at least 3 webcams. We propose an additional video stream management that extends the number of webcams to 10 or more. These considerations make our system low-cost and hence accessible for everyone. We also present an adaptation of our plane-sweep method to create simultaneously multiple views of the scene in real-time. Our system is especially designed for stereovision using autostereoscopic displays. The new views are computed from 4 webcams connected to a computer and are compressed in order to be transfered to a mobile phone. Using GPU programming, our method provides up to 16 images of the scene in real-time. The use of both GPU and CPU makes this method work on only one consumer grade computer.
基金Supported by the National Natural Science Foundation of China (No. 60672073,60872094)the Program for New Century Excellent Talents in University (NCET-06-0537)the Natural Science Foundation of Ningbo (No. 2007A610037).
文摘A semi-reference image quality assessment metric based on similarity measurement for synthesized virtual viewpoint image （VVI） in free-viewpoint television system （FFV） is proposed in this paper. The key point of the proposed metric is taking resemblant information between VVI and its neighbor view images for quality assessment to make our metric to be extended to multi-semi-reference image quality assessment easily. The proposed metric first extracts impact factors from image features, then combines an image synthesis technique and similarity functions, in which, disparity information are taken into account for registering the resemblant regions. Experiments are divided into three phases. Phase I is to verify the validation of the proposed metric by taking impaired images and original reference into account. The experimental results show the agreement between evaluation scores and bio-characteristic of human visual system. Phase II shows the accordance with Phase I by taking neighbor view as reference. The proposed metric can be taken as a full reference one to evaluate the image quality even though the original reference is absent. Phase III is then performed to evaluate the quality of WI. Evaluation scores in the experimental results are able to evaluate the quality of VVI.
基金supported by the National Natural Science Foundation of China(Nos.62172315,62073262,and 61672429)the Fundamental Research Funds for the Central Universities,the Innovation Fund of Xidian University(No.20109205456)the Key Research and Development Program of Shaanxi(No.S2021-YF-ZDCXL-ZDLGY-0127),and HUAWEI.
文摘Free-viewpoint video allows the user to view objects from any virtual perspective,creating an immersive visual experience.This technology enhances the interactivity and freedom of multimedia performances.However,many free-viewpoint video synthesis methods hardly satisfy the requirement to work in real time with high precision,particularly for sports fields having large areas and numerous moving objects.To address these issues,we propose a freeviewpoint video synthesis method based on distance field acceleration.The central idea is to fuse multiview distance field information and use it to adjust the search step size adaptively.Adaptive step size search is used in two ways:for fast estimation of multiobject three-dimensional surfaces,and synthetic view rendering based on global occlusion judgement.We have implemented our ideas using parallel computing for interactive display,using CUDA and OpenGL frameworks,and have used real-world and simulated experimental datasets for evaluation.The results show that the proposed method can render free-viewpoint videos with multiple objects on large sports fields at 25 fps.Furthermore,the visual quality of our synthetic novel viewpoint images exceeds that of state-of-the-art neural-rendering-based methods.
基金supported by the Basic Science Research Program through the National Research Foundation of Korea(NRF)funded by the Ministry of Education(NRF-2019R1F1A1061635)by a research grant from Seoul Women’s University(2020-0213).
文摘The trend in video viewing has been evolving beyond simply providing a multi-view option.Recently,a function that allows selection and viewing of a clip from a multi-view service that captures a specific range or object has been added.In particular,the free-view service is an extended concept of multi-view and provides a freer viewpoint.However,since numerous videos and additional data are required for its construction,all of the clips constituting the content cannot be simultaneously provided.Only certain clips are selected and provided to the user.If the video is not the preferred video,change request is made,and a delay occurs during retransmission from the server.Delays due to frequent re-requests degrade the overall quality of service.For free-view services,selectively transmitting the video according to the user’s desired viewpoint and region of interest within the limited network of available videos is important.In this study,we propose a method of screening and providing the correct video based on objects in the contents.Based on the method of recognizing the object in each clip,we designed a method of setting its priority based on information about the object’s location for each viewpoint.During the transmission and receiving process using this information,the selected video can be rapidly recognized and changed.Herein,we present a service system configuration method and propose video selection examples for free-view services.
基金supported by the National Natural Science Foundation of China(Grant No.60832003)Key Laboratory of Advanced Display and System Application(Shanghai University),Ministry of Education,China(Grant No.P200902)the Key Project of Science and Technology Commission of Shanghai Municipality(Grant No.10510500500)
文摘Depth maps are used for synthesis virtual view in free-viewpoint television （FTV） systems. When depth maps are derived using existing depth estimation methods, the depth distortions will cause undesirable artifacts in the synthesized views. To solve this problem, a 3D video quality model base depth maps （D-3DV） for virtual view synthesis and depth map coding in the FTV applications is proposed. First, the relationships between distortions in coded depth map and rendered view are derived. Then, a precisely 3DV quality model based depth characteristics is develop for the synthesized virtual views. Finally, based on D-3DV model, a multilateral filtering is applied as a pre-processed filter to reduce rendering artifacts. The experimental results evaluated by objective and subjective methods indicate that the proposed D-3DV model can reduce bit-rate of depth coding and achieve better rendering quality.
