Depth Map Prediction of Occluded Objects Using Structure Tensor with Gain Regularization

The creation of the 3D rendering model involves the prediction of an accurate depth map for the input images.A proposed approach of a modified semi-global block matching algorithm with variable window size and the gradient assessment of objects predicts the depth map.3D modeling and view synthesis algorithms could effectively handle the obtained disparity maps.This work uses the consistency check method to find an accurate depth map for identifying occluded pixels.The prediction of the disparity map by semi-global block matching has used the benchmark dataset of Middlebury stereo for evaluation.The improved depth map quality within a reasonable process-ing time outperforms the other existing depth map prediction algorithms.The experimental results have shown that the proposed depth map predictioncould identify the inter-object boundaryeven with the presence ofocclusion with less detection error and runtime.We observed that the Middlebury stereo dataset has very few images with occluded objects,which made the attainment of gain cumbersome.Considering this gain,we have created our dataset with occlu-sion using the structured lighting technique.The proposed regularization term as an optimization process in the graph cut algorithm handles occlusion for different smoothing coefficients.The experimented results demonstrated that our dataset had outperformed the Tsukuba dataset regarding the percentage of occluded pixels.

基于图像关键帧的Visual-Depth Map建立方法

对于室内视觉定位系统,需要在离线阶段建立Visual Map数据库用来存储图像信息,在线阶段用户通过与Visual Map数据库进行比对来完成用户位置的估计.离线阶段建立的数据库可以采用逐点采样或视频流采样的方式.但是无论何种方式,考虑到数据库中图像信息的相似性,传统方式建立的数据库中存储图像有较多冗余,导致增加了在线阶段的定位时间开销.因此,本文根据Visual Map中的相邻图像间的相似性,提出了一种基于图像关键帧的Visual-Depth Map建立方法,有效地减少了离线数据库的规模.在离线阶段,本文使用Kinect传感器同时获得图像信息和深度信息;然后,通过基于图像相似度的图像关键帧算法对原始图像序列进行筛选,得到关键帧序列,从而实现Visual-Depth Map的建立.在线阶段,用户可以直接输入查询图像与Visual-Depth Map中的图像序列进行检索匹配,找到相似度较高的匹配图像,再通过EPnP算法进行2D-3D的位姿估计,完成用户位置的计算.实验证明,本文所提方法可以在保证较高定位精度的前提下,有效减少离线数据库规模,降低在线阶段的定位时间开销.

INPAINTING ALGORITHM FOR KINECT DEPTH MAP BASED ON FOREGROUND SEGMENTATION

The depth information of the scene indicates the distance between the object and the camera,and depth extraction is a key technology in 3D video system.The emergence of Kinect makes the high resolution depth map capturing possible.However,the depth map captured by Kinect can not be directly used due to the existing holes and noises,which needs to be repaired.We propose a texture combined inpainting algorithm in this paper.Firstly,the foreground is segmented combined with the color characteristics of the texture image to repair the foreground of the depth map.Secondly,region growing is used to determine the match region of the hole in the depth map,and to accurately position the match region according to the texture information.Then the match region is weighted to fill the hole.Finally,a Gaussian filter is used to remove the noise in the depth map.Experimental results show that the proposed method can effectively repair the holes existing in the original depth map and get an accurate and smooth depth map,which can be used to render a virtual image with good quality.

Technique of Embedding Depth Maps into 2D Images

This paper proposes a new technique that is used to embed depth maps into corresponding 2-dimensional （2D） images. Since a 2D image and its depth map are integrated into one type of image format, they can be treated as if they were one 2D image. Thereby, it can reduce the amount of data in 3D images by half and simplify the processes for sending them through networks because the synchronization between images for the left and right eyes becomes unnecessary. We embed depth maps in the quantized discrete cosine transform （DCT） data of 2D images. The key to this technique is whether the depth maps could be embedded into 2D images without perceivably deteriorating their quality. We try to reduce their deterioration by compressing the depth map data by using the differences from the next pixel to the left. We assume that there is only one non-zero pixel at most on one horizontal line in the DCT block because the depth map values change abruptly. We conduct an experiment to evaluate the quality of the 2D images embedded with depth maps and find that satisfactory quality could be achieved.

3DV quality model based depth maps for view synthesis in FTV system

Depth maps are used for synthesis virtual view in free-viewpoint television （FTV） systems. When depth maps are derived using existing depth estimation methods, the depth distortions will cause undesirable artifacts in the synthesized views. To solve this problem, a 3D video quality model base depth maps （D-3DV） for virtual view synthesis and depth map coding in the FTV applications is proposed. First, the relationships between distortions in coded depth map and rendered view are derived. Then, a precisely 3DV quality model based depth characteristics is develop for the synthesized virtual views. Finally, based on D-3DV model, a multilateral filtering is applied as a pre-processed filter to reduce rendering artifacts. The experimental results evaluated by objective and subjective methods indicate that the proposed D-3DV model can reduce bit-rate of depth coding and achieve better rendering quality.

Depth map软件在网师园空间分析中的应用

空间句法是一种通过对包括建筑、聚落、城市以及景观在内的人居空间结构的量化描述来研究空间组织的理论和方法。作为空间句法的专用分析软件,近年来Depth map在空间结构分析领域得到了越来越广泛的应用。以江南园林网师园为例,利用Depth map软件建立模型,对其空间的可视和可行2个层面分别进行连接度、整合度和深度值的空间结构分析,并得出相应结论:Depth map在定量分析园林空间特性,确定园林空间不同层面之间的联系方面起着重要作用。

An integrated method of selecting environmental covariates for predictive soil depth mapping

Environmental covariates are the basis of predictive soil mapping.Their selection determines the performance of soil mapping to a great extent,especially in cases where the number of soil samples is limited but soil spatial heterogeneity is high.In this study,we proposed an integrated method to select environmental covariates for predictive soil depth mapping.First,candidate variables that may influence the development of soil depth were selected based on pedogenetic knowledge.Second,three conventional methods(Pearson correlation analysis(PsCA),generalized additive models(GAMs),and Random Forest(RF))were used to generate optimal combinations of environmental covariates.Finally,three optimal combinations were integrated to produce a final combination based on the importance and occurrence frequency of each environmental covariate.We tested this method for soil depth mapping in the upper reaches of the Heihe River Basin in Northwest China.A total of 129 soil sampling sites were collected using a representative sampling strategy,and RF and support vector machine(SVM)models were used to map soil depth.The results showed that compared to the set of environmental covariates selected by the three conventional selection methods,the set of environmental covariates selected by the proposed method achieved higher mapping accuracy.The combination from the proposed method obtained a root mean square error(RMSE)of 11.88 cm,which was 2.25–7.64 cm lower than the other methods,and an R^2 value of 0.76,which was 0.08–0.26 higher than the other methods.The results suggest that our method can be used as an alternative to the conventional methods for soil depth mapping and may also be effective for mapping other soil properties.

Edge-Preserving Depth Map Super-Resolution with Intensity Guidance

Depth discontinuity edge affects the visual quality of synthesized images in 3D image warping.However,it suffers from accuracy degradation when up-sampled from low-resolution depth maps,especially at large scaling factors.To preserve the accuracy of depth discontinuity,a novel joint bilateral depth super-resolution with intensity guidance method is proposed.Particularly,the fast local intensity classification is exploited to estimate depth coefficients in joint bilateral up-sampling for depth maps,so as to eliminate depth discontinuity edge misalignment.Additionally,the proposed method is accelerated on graphic processing units(GPUs)to meet the requirement of realtime application.Experiments demonstrate that our method can preserve the accuracy of depth discontinuity edges after super resolution,leveraging the visual quality of synthesized image in 3D image warping.

DGConv: A Novel Convolutional Neural Network Approach for Weld Seam Depth Image Detection

We propose a novel image segmentation algorithm to tackle the challenge of limited recognition and segmentation performance in identifying welding seam images during robotic intelligent operations.Initially,to enhance the capability of deep neural networks in extracting geometric attributes from depth images,we developed a novel deep geometric convolution operator(DGConv).DGConv is utilized to construct a deep local geometric feature extraction module,facilitating a more comprehensive exploration of the intrinsic geometric information within depth images.Secondly,we integrate the newly proposed deep geometric feature module with the Fully Convolutional Network(FCN8)to establish a high-performance deep neural network algorithm tailored for depth image segmentation.Concurrently,we enhance the FCN8 detection head by separating the segmentation and classification processes.This enhancement significantly boosts the network’s overall detection capability.Thirdly,for a comprehensive assessment of our proposed algorithm and its applicability in real-world industrial settings,we curated a line-scan image dataset featuring weld seams.This dataset,named the Standardized Linear Depth Profile(SLDP)dataset,was collected from actual industrial sites where autonomous robots are in operation.Ultimately,we conducted experiments utilizing the SLDP dataset,achieving an average accuracy of 92.7%.Our proposed approach exhibited a remarkable performance improvement over the prior method on the identical dataset.Moreover,we have successfully deployed the proposed algorithm in genuine industrial environments,fulfilling the prerequisites of unmanned robot operations.

K-nearest neighborhood based integration of time-of-flight cameras and passive stereo for high-accuracy depth maps?

Both time-of-flight(ToF) cameras and passive stereo can provide the depth information for their corresponding captured real scenes, but they have innate limitations. ToF cameras and passive stereo are intrinsically complementary for certain tasks. It is desirable to appropriately leverage all the available information by ToF cameras and passive stereo. Although some fusion methods have been presented recently, they fail to consider ToF reliability detection and ToF based improvement of passive stereo. As a result, this study proposes an approach to integrating ToF cameras and passive stereo to obtain high-accuracy depth maps. The main contributions are:(1) An energy cost function is devised to use data from ToF cameras to boost the stereo matching of passive stereo;(2) A fusion method is used to combine the depth information from both ToF cameras and passive stereo to obtain high-accuracy depth maps. Experiments show that the proposed approach achieves improved results with high accuracy and robustness.

Depth map resolution improvement for 3D range-intensity correlation laser imaging

This Letter proposes a high bit-depth coding method to improve depth map resolution and render it suitable to human-eye observation in 3D range-intensity correlation laser imaging. In this method, a high bit-depth CCD camera with a nanosecond-sealed gated intensifier is used as an image sensor; subsequently two high bit-depth gate images with specific range-intensity profiles are obtained to establish the gray depth map and finally the gray depth map is encoded by an equidensity pseudocolor. With this method, a color depth map is generated with higher range resolution. In our experimental work, the range resolution of the depth map is improved by a factor of 1.67.

Consistent Depth Maps Estimation from Binocular Stereo Video Sequence

In the paper, an approach is proposed for the problem of consistency in depth maps estimation from binocular stereo video sequence. The consistent method includes temporal consistency and spatial consistency to eliminate the flickering artifacts and smooth inaccuracy in depth recovery. So the improved global stereo matching based on graph cut and energy optimization is implemented. In temporal domain, the penalty function with coherence factor is introduced for temporal consistency, and the factor is determined by Lucas-Kanade optical flow weighted histogram similarity constraint(LKWHSC). In spatial domain, the joint bilateral truncated absolute difference(JBTAD) is proposed for segmentation smoothing. The method can smooth naturally and uniformly in low-gradient region and avoid over-smoothing as well as keep edge sharpness in high-gradient discontinuities to realize spatial consistency. The experimental results show that the algorithm can obtain better spatial and temporal consistent depth maps compared with the existing algorithms.

Interactive lighting editing system for single indoor low-light scene images with corresponding depth maps

We propose a novel interactive lighting editing system for lighting a single indoor RGB image based on spherical harmonic lighting.It allows users to intuitively edit illumination and relight the complicated low-light indoor scene.Our method not only achieves plausible global relighting but also enhances the local details of the complicated scene according to the spatially-varying spherical harmonic lighting,which only requires a single RGB image along with a corresponding depth map.To this end,we first present a joint optimization algorithm,which is based on the geometric optimization of the depth map and intrinsic image decomposition avoiding texture-copy,for refining the depth map and obtaining the shading map.Then we propose a lighting estimation method based on spherical harmonic lighting,which not only achieves the global illumination estimation of the scene,but also further enhances local details of the complicated scene.Finally,we use a simple and intuitive interactive method to edit the environment lighting map to adjust lighting and relight the scene.Through extensive experimental results,we demonstrate that our proposed approach is simple and intuitive for relighting the low-light indoor scene,and achieve state-of-the-art results.

Performance study of feature descriptors for human detection on depth map

Depth map contains the space information of objects and is almost free from the influence of light,and it attracts many research interests in the field of machine vision used for human detection.Therefore,hunting a suitable image feature for human detection on depth map is rather attractive.In this paper,we evaluate the performance of the typical features on depth map.A depth map dataset containing various indoor scenes with human is constructed by using Microsoft’s Kinect camera as a quantitative benchmark for the study of methods of human detection on depth map.The depth map is smoothed with pixel filtering and context filtering so as to reduce particulate noise.Then,the performance of five image features and a new feature is studied and compared for human detection on the dataset through theoretic analysis and simulation experiments.Results show that the new feature outperforms other descriptors.

基于depth-map和分布式视频编码的多视点视频传输方法

针对多视点视频传输系统数据量庞大的问题,提出一种基于深度图(depth-map)和分布式视频编码(DVC)的不等错误保护(UEP)传输方法。该方法首先基于多个视点提取深度图;然后,在传输过程中传输一个视点及其深度图;最后,经过网络传输,在解码端由一个视点图及其深度图生成其他视点。由于视点图和深度图在解码端的重要程度不同,对需要传输的视点图和深度图采用不同的分布式视频编码方法,进行不平等的错误保护。仿真实验结果表明,所提传输方法比传统的分布式多视点视频编码传输系统具有更好的抗误码性能,提高了传输可靠性,图像的峰值信噪比(PSNR)约提高1.5 dB。

A Fast Depth-Map Generation Algorithm based on Motion Search from 2D Video Contents

Generation of a depth-map from 2D video is the kernel of DIBR (Depth Image Based Rendering) in 2D-3D video conversion systems. However it occupies over most of the system resource where the motion search module takes up 90% time-consuming in typical motion estimation-based depth-map generation algorithms. In order to reduce the computational complexity, in this paper a new fast depth-map generation algorithm based on motion search is developed, in which a fast diamond search algorithm is adopted to decide whether a 16x16 or 4x4 block size is used based on Sobel operator in the motion search module to obtain a sub-depth-map. Then the sub-depth-map will be fused with the sub-depth-maps gotten from depth from color component Cr and depth from linear perspective modules to compensate and refine detail of the depth-map, finally obtain a better depth-map. The simulation results demonstrate that the new approach can greatly reduce over 50% computational complexity compared to other existing methods.

Combining Multi-scale Directed Depth Motion Maps and Log-Gabor Filters for Human Action Recognition

Recognition of the human actions by computer vision has become an active research area in recent years. Due to the speed and the high similarity of the actions, the current algorithms cannot get high recognition rate. A new recognition method of the human action is proposed with the multi-scale directed depth motion maps(MsdDMMs) and Log-Gabor filters. According to the difference between the speed and time order of an action, MsdDMMs is proposed under the energy framework. Meanwhile, Log-Gabor is utilized to describe the texture details of MsdDMMs for the motion characteristics. It can easily satisfy both the texture characterization and the visual features of human eye. Furthermore, the collaborative representation is employed as action recognition by the classification. Experimental results show that the proposed algorithm, which is applied in the MSRAction3 D dataset and MSRGesture3 D dataset, can achieve the accuracy of 95.79% and 96.43% respectively. It also has higher accuracy than the existing algorithms, such as super normal vector(SNV), hierarchical recurrent neural network(Hierarchical RNN).

Generate Reservoir Depths Mapping by Using Digital Elevation Model: A Case Study of Mosul Dam Lake, Northern Iraq

Not many bathymetric maps are available for many lakes and reservoirs in developing countries. Usually the bathymetric mapping requires investment in expensive equipment and fieldwork, both of which are not accessible in these countries. This work demonstrates the ability to develop bathymetric map of Mosul Lake by using a digital elevation model (DEM). The depths model of the lake was designed through the use of three main stages;a coastline extraction, dataset interpolation and a triangular irregular network model. The normalized difference water index (NDWI) was used for automatic delineation of the lake coastline from satellite images. The ordinary kriging interpolation with a stable model was used to interpolate the water depths dataset. Finally a triangulated irregular network (TIN) model was used to visualize the resulting interpolation model. Calculated values of area and volume of a TIN model during 2011 were compared with values of supposed initial operation of the reservoir. The differences of water volume storage between these stages at 321 m water level was about 0.81 × 109 m3, where the lake lost around 10% of storage value. Also the results of depths lake model show that the change in water storage between March and July 2011 was about 3.08 × 109 m3.

基于锚点的快速三维手部关键点检测算法

在人机协作任务中,手部关键点检测为机械臂提供目标点坐标,A2J(Anchor-to-Joint)是具有代表性的一种利用锚点进行关键点检测的方法。A2J以深度图为输入,可实现较好的检测效果,但对全局特征获取能力不足。文中设计了全局-局部特征融合模块(Global-Local Feature Fusion,GLFF)对骨干网络浅层和深层的特征进行融合。为了提升检测速度,文中将A2J的骨干网络替换为ShuffleNetv2并对其进行改造,用5×5深度可分离卷积替换3×3深度可分离卷积,增大感受野,有效提升了骨干网络对全局特征的提取能力。文中在锚点权重估计分支引入高效通道注意力模块(Efficient Channel Attention,ECA),提升了网络对重要锚点的关注度。在主流数据集ICVL和NYU上进行的训练和测试结果表明,相比于A2J,文中所提方法的平均误差分别降低了0.09 mm和0.15 mm。在GTX1080Ti显卡上实现了151 frame·s^(-1)的检测速率,满足人机协作任务对于实时性的要求。

基于映射空间编码的高速运动轨道图像去模糊研究

针对轨道缺陷检测系统因镜头抖动或相机快速移动而导致所采集图像较为模糊的问题,提出一种基于最大后验概率估计思想的映射空间编码的高速运动轨道图像去模糊算法。首先,该算法使用深度编解码器和残差网络分别对数据集中清晰图像到模糊图像的映射关系和模糊核进行编码,为了保证编码时频率信息的完整性,算法在传统的残差模块上引入快速傅里叶变换通道构成双通道残差网络,以补偿多次特征提取带来的频率损失;其次,算法采用深度图像先验(Deep Image Prior,DIP)将潜在的清晰图像和模糊核进行参数化,再利用先验得到的模糊核和清晰图像来调用编码空间中的映射关系;最后,通过交替优化潜在的清晰图像和模糊核,从而去逼近一个真实未知的映射,进而实现真实场景下高速运动轨道图像的去模糊。实验结果表明,双通道残差模块提取的特征图频率信息分量强度普遍高于传统的残差模块,相较于使用传统残差模块实现该算法,采用双通道残差模块可使峰值信噪比(Peak Signal-to-Noise Ratio,PSNR)提升0.84 dB,结构相似性(Structural Similarity,SSIM)提高0.025 1。与现有的深度学习去模糊算法相比,提出的去模糊算法对高速轨道检测系统所采集图像的去模糊效果更佳,在性能方面相较于最好的去模糊算法,PSNR提高了1.84 dB,SSIM提升了0.017 3,显著提升了采集图像的质量。研究结果可为下一步识别轨道部件是否存在缺陷提供清晰图像。