Survey of 3D modeling using depth cameras

Three-dimensional(3D)modeling is an important topic in computer graphics and computer vision.In recent years,the introduction of consumer-grade depth cameras has resulted in profound advances in 3D modeling.Starting with the basic data structure,this survey reviews the latest developments of 3D modeling based on depth cameras,including research works on camera tracking,3D object and scene reconstruction,and high-quality texture reconstruction.We also discuss the future work and possible solutions for 3D modeling based on the depth camera.

Depth error correction for projector-camera based consumer depth cameras

This paper proposes a depth measurement error model for consumer depth cameras such as the Microsoft Kinect, and a corresponding calibration method. These devices were originally designed as video game interfaces, and their output depth maps usually lack sufficient accuracy for 3 D measurement.Models have been proposed to reduce these depth errors, but they only consider camera-related causes.Since the depth sensors are based on projectorcamera systems, we should also consider projectorrelated causes. Also, previous models require disparity observations, which are usually not output by such sensors, so cannot be employed in practice. We give an alternative error model for projector-camera based consumer depth cameras, based on their depth measurement algorithm, and intrinsic parameters of the camera and the projector; it does not need disparity values. We also give a corresponding new parameter estimation method which simply needs observation of a planar board. Our calibrated error model allows use of a consumer depth sensor as a 3 D measuring device.Experimental results show the validity and effectiveness of the error model and calibration procedure.

Markerless Respiratory Motion Tracking Using Single Depth Camera

The aim of this study is to propose a novel system that has an ability to detect intra-fractional motion during radiotherapy treatment in real-time using three-dimensional surface taken by a depth camera, Microsoft Kinect v1. Our approach introduces three new aspects for three-dimensional surface tracking in radiotherapy treatment. The first aspect is a new algorithm for noise reduction of depth values. Ueda’s algorithm was implemented and enabling a fast least square regression of depth values. The second aspect is an application for detection of patient’s motion at multiple points in thracoabdominal regions. The third aspect is an estimation of three-dimensional surface from multiple depth values. For evaluation of noise reduction by Ueda’s algorithm, two respiratory patterns are measured by the Kinect as well as a laser range meter. The resulting cross correlation coefficients between the laser range meter and the Kinect were 0.982 for abdominal respiration and 0.995 for breath holding. Moreover, the mean cross correlation coefficients between the signals of our system and the signals of Anzai with respect to participant’s respiratory motion were 0.90 for thoracic respiration and 0.93 for abdominal respiration, respectively. These results proved that the performance of the developed system was comparable to existing motion monitoring devices. Reconstruction of three-dimensional surface also enabled us to detect the irregular motion and breathing arrest by comparing the averaged depth with predefined threshold values.

A Linear Approach for Depth and Colour Camera Calibration Using Hybrid Parameters

Many recent applications of computer graphics and human computer interaction have adopted both colour cameras and depth cameras as input devices. Therefore, an effective calibration of both types of hardware taking different colour and depth inputs is required. Our approach removes the numerical difficulties of using non-linear optimization in previous methods which explicitly resolve camera intrinsics as well as the transformation between depth and colour cameras. A matrix of hybrid parameters is introduced to linearize our optimization. The hybrid parameters offer a transformation from a depth parametric space （depth camera image） to a colour parametric space （colour camera image） by combining the intrinsic parameters of depth camera and a rotation transformation from depth camera to colour camera. Both the rotation transformation and intrinsic parameters can be explicitly calculated from our hybrid parameters with the help of a standard QR factorisation. We test our algorithm with both synthesized data and real-world data where ground-truth depth information is captured by Microsoft Kinect. The experiments show that our approach can provide comparable accuracy of calibration with the state-of-the-art algorithms while taking much less computation time （1/50 of Herrera＇s method and 1/10 of Raposo＇s method） due to the advantage of using hybrid parameters.

Method for C/N ratio estimation using Mask R-CNN and a depth camera for organic fraction of municipal solid wastes

Fast assessment of the initial carbon to nitrogen ratio(C/N)of organic fraction of municipal solid waste(OFMSW)is an important prerequisite for automatic composting control to improve efficiency and stability of the bioconversion process.In this study,a novel approach was proposed to estimate the C/N of OFMSW,where an instance segmentation model was applied to predict the masks for the waste images.Then,by combining the instance segmentation model with the depth-camera-based volume calculation algorithm,the volumes occupied by each type of waste were obtained,therefore the C/N could be estimated based on the properties of each type of waste.First,an instance segmentation dataset including three common classes of OFMSW was built to train mask region-based convolutional neural networks(Mask R-CNN)model.Second,a volume measurement algorithm was proposed,where the measurement result of the object was derived by accumulating the volumes of small rectangular cuboids whose bottom area was calculated with the projection property.Then the calculated volume was corrected with linear regression models.The results showed that the trained instance segmentation model performed well with average precision scores AP_(50)=82.9,AP_(75)=72.5,and mask intersection over unit(Mask IoU)=45.1.A high correlation was found between the estimated C/N and the ground truth with a coefficient of determination R2=0.97 and root mean square error RMSE=0.10.The relative average error was 0.42%and the maximum error was only 1.71%,which indicated this approach has potential for practical applications.

Depth Camera-Based Robot-Assisted Ultrasonic Lipolysis System

With many advantages such as non-invasive,safe and quick effect,focused ultrasound lipolysis stands out among many fat-removing methods.However,during the whole process,the doctor needs to hold the ultrasound transducer and press it on the patient’s skin with a large pressure for a long time;thus the probability of muscle and bone damage for doctors is greatly increased.To reduce the occurrence of doctors’occupational diseases,a depth camera-based ultrasonic lipolysis robot system is proposed to realize robot-assisted automatic ultrasonic lipolysis operation.The system is composed of RealSense depth camera,KUKA LBR Med seven-axis robotic arm,PC host,and ultrasonic lipolysis instrument.The whole operation includes two parts:preoperative planning and intraoperative operation.In preoperative planning,the treatment area is selected in the camera image by the doctor;then the system automatically plans uniformly distributed treatment points in the treatment area.At the same time,the skin normal vector is calculated to determine the end posture of the robot,so that the ultrasound transducer can be pressed down in the normal direction of skin.During the intraoperative operation,the robot is controlled to arrive at the treatment point in turn.Meanwhile,the patient’s movement can be detected by the depth camera,and the path of robot is adjusted in real time so that the robot can track the movement of patient,thereby ensuring the accuracy of the ultrasonic lipolysis operation.Finally,the human body model experiment is conducted.The results show that the maximum error of the robot operation is within 5mm,average error is 3.1mm,and the treatment points of the robot operation are more uniform than those of manual operation.Therefore,the system can replace the doctor and achieve autonomous ultrasonic lipolysis to reduce the doctor’s labor intensity.

ViT-based Terrain Recognition System for wearable soft exosuit

Terrain classification and force assistance strategies in complex environments have always piqued the interest of many researchers.For wearable soft exosuits,inaccurate terrain recognition can easily introduce undesired assist forces that can easily injure the wearer.Because of these problems,we introduced a depth camera into the exosuit system,perform classification of terrain based on a Vision Transformer(ViT),and optimized the control algorithm,which is known as a ViT-Based Terrain Recognition System(TTRS).First,we used the Transformer algorithm to achieve a considerable classification effect in terrain recognition.We also introduced terrain recognition as prior knowledge into the force assistance strategy of the exosuit,providing different force assistance to the exosuit in different terrains.Subsequently,we performed human experiments with seven able-bodied people(six males and one female).The promising results demonstrate that our classification accuracy can reach 99.2%under six different terrains and that it can smoothly switch the force–assist curve in different terrains to better adapt to the complex terrain and improve the walking effect.The aforementioned terrain recognition algorithms and force–assist strategies may positively influence the study of soft exosuit,powered prostheses,and orthotics.