Application of Opening and Closing Morphology in Deep Learning-Based Brain Image Registration

In order to improve the registration accuracy of brain magnetic resonance images(MRI),some deep learning registration methods use segmentation images for training model.How-ever,the segmentation values are constant for each label,which leads to the gradient variation con-centrating on the boundary.Thus,the dense deformation field(DDF)is gathered on the boundary and there even appears folding phenomenon.In order to fully leverage the label information,the morphological opening and closing information maps are introduced to enlarge the non-zero gradi-ent regions and improve the accuracy of DDF estimation.The opening information maps supervise the registration model to focus on smaller,narrow brain regions.The closing information maps supervise the registration model to pay more attention to the complex boundary region.Then,opening and closing morphology networks(OC_Net)are designed to automatically generate open-ing and closing information maps to realize the end-to-end training process.Finally,a new registra-tion architecture,VM_(seg+oc),is proposed by combining OC_Net and VoxelMorph.Experimental results show that the registration accuracy of VM_(seg+oc) is significantly improved on LPBA40 and OASIS1 datasets.Especially,VM_(seg+oc) can well improve registration accuracy in smaller brain regions and narrow regions.

RANSACs for 3D Rigid Registration:A Comparative Evaluation

Estimating an accurate six-degree-of-freedom(6-Do F)pose from correspondences with outliers remains a critical issue to 3D rigid registration.Random sample consensus(RANSAC)and its variants are popular solutions to this problem.Although there have been a number of RANSAC-fashion estimators,two issues remain unsolved.First,it is unclear which estimator is more appropriate to a particular application.Second,the impacts of different sampling strategies,hypothesis generation methods,hypothesis evaluation metrics,and stop criteria on the overall estimators remain ambiguous.This work fills these gaps by first considering six existing RANSAC-fashion methods and then proposing eight variants for a comprehensive evaluation.The objective is to thoroughly compare estimators in the RANSAC family,and evaluate the effects of each key stage on the eventual 6-Do F pose estimation performance.Experiments have been carried out on four standard datasets with different application scenarios,data modalities,and nuisances.They provide us with input correspondence sets with a variety of inlier ratios,spatial distributions,and scales.Based on the experimental results,we summarize remarkable outcomes and valuable findings,so as to give practical instructions to real-world applications,and highlight current bottlenecks and potential solutions in this research realm.

2D-3D registration for 3D analysis of lower limb alignment in a weight-bearing condition

X-ray imaging is the conventional method for diagnosing the orthopedic condition of a patient. Computerized Tomography(CT) scanning is another diagnostic method that provides patient’s 3D anatomical information. However, both methods have limitations when diagnosing the whole leg; X-ray imaging does not provide 3D information, and normal CT scanning cannot be performed with a standing posture. Obtaining 3D data regarding the whole leg in a standing posture is clinically important because it enables 3D analysis in the weight bearing condition.Based on these clinical needs, a hardware-based bi-plane X-ray imaging system has been developed; it uses two orthogonal X-ray images. However, such methods have not been made available in general clinics because of the hight cost. Therefore, we proposed a widely adaptive method for 2 D X-ray image and 3D CT scan data. By this method, it is possible to threedimensionally analyze the whole leg in standing posture. The optimal position that generates the most similar image is the captured X-ray image. The algorithm verifies the similarity using the performance of the proposed method by simulation-based experiments. Then, we analyzed the internal-external rotation angle of the femur using real patient data. Approximately 10.55 degrees of internal rotations were found relative to the defined anterior-posterior direction. In this paper, we present a useful registration method using the conventional X-ray image and 3D CT scan data to analyze the whole leg in the weight-bearing condition.

A Laplacian Surface Deformation and Optimization Based 3D Registration Algorithm for Image Guided Prostate Radiotherapy

Purpose: To develop a fast landmark-based deformable registration method to capture the soft tissue transformation between the planning 3D CT images and treatment 3D cone-beam CT (CBCT) images for the adaptive external beam radiotherapy (EBRT). Method and Materials: The developed method was based on a global-to-local landmark-based deformable registration algorithm. The landmarks were first acquired by applying a fast segmentation method using the active shape model. The global registration method was applied to establish a registration framework. The Laplacian surface deformation (LSD) and Laplacian surface optimization (LSO) method were then employed for local deformation and remeshing respectively to reach an optimal registration solution. In LSD, the deformed mesh is generated by minimizing the quadratic energy to keep the shape and to move control points to the target position. In LSO, a mesh is reconstructed by minimizing the quadratic energy to smooth the object by minimizing the difference while keeping the landmarks unchanged. The method was applied on 8 EBRT prostate datasets. The distance and volume based estimators were used to evaluate the results. The target volumes delineated by physicians were used as gold standards in the evaluation. Results: The entire segmentation and registration processing time was within 1 minute for all the datasets. The mean distance estimators ranged from 0.43 mm to 2.23 mm for the corresponding model points between the treatment CBCT images and the registered planning images. The mean overlap ratio ranged from 85.5% to 93.2% of the prostate volumes after registration. These results demonstrated reasonably good agreement between the developed method and the gold standards. Conclusion: A novel and fast landmark-based deformable registration method is developed to capture the soft tissue transformation between the planning and treatment images for prostate target volumes. The results show that with the method the image registration and transformation can be completed within one minute and has the potential to be applied to real-time adaptive radiotherapy.

A new projection model based robust 2D-3D registration method on Fourier-Mellin space for image guided intervention

An automatic method is proposed to solve the registration problem,which aligns a single 2D fluoroscopic image to a 3D image volume without demanding any additional media like calibration plate or user interactions.First,a mathematic projection model is designed which can reduce the influence of projection distortion on parameter optimization and improve the registration accuracy.Then,a two stage optimization method is proposed,which enables a robust registration in a wide parameter space.Furthermore,an automatic registration framework is proposed based on the FourierMellin robust image comparison descriptor.Experimental results show that the registration method has a high accuracy with average rotation error of 0.6 degree and average translation error of 1.4mm.

Fast Estimation of Loader’s Shovel Load Volume by 3D Reconstruction of Material Piles

Fast and accurate measurement of the volume of earthmoving materials is of great signifcance for the real-time evaluation of loader operation efciency and the realization of autonomous operation. Existing methods for volume measurement, such as total station-based methods, cannot measure the volume in real time, while the bucket-based method also has the disadvantage of poor universality. In this study, a fast estimation method for a loader’s shovel load volume by 3D reconstruction of material piles is proposed. First, a dense stereo matching method (QORB–MAPM) was proposed by integrating the improved quadtree ORB algorithm (QORB) and the maximum a posteriori probability model (MAPM), which achieves fast matching of feature points and dense 3D reconstruction of material piles. Second, the 3D point cloud model of the material piles before and after shoveling was registered and segmented to obtain the 3D point cloud model of the shoveling area, and the Alpha-shape algorithm of Delaunay triangulation was used to estimate the volume of the 3D point cloud model. Finally, a shovel loading volume measurement experiment was conducted under loose-soil working conditions. The results show that the shovel loading volume estimation method (QORB–MAPM VE) proposed in this study has higher estimation accuracy and less calculation time in volume estimation and bucket fll factor estimation, and it has signifcant theoretical research and engineering application value.

Multi-View Point-Based Registration for Native Knee Kinematics Measurement with Feature Transfer Learning

Deep-learning methods provide a promising approach for measuring in-vivo knee joint motion from fast registration of two-dimensional(2D)to three-dimensional(3D)data with a broad range of capture.However,if there are insufficient data for training,the data-driven approach will fail.We propose a feature-based transfer-learning method to extract features from fluoroscopic images.With three subjects and fewer than 100 pairs of real fluoroscopic images,we achieved a mean registration success rate of up to 40%.The proposed method provides a promising solution,using a learning-based registration method when only a limited number of real fluoroscopic images is available.

Balanced Functional Maps for Three-Dimensional Non-Rigid Shape Registration

Three-dimensional(3D)shape registration is a challenging problem,especially for shapes under non-rigid transformations.In this paper,a 3D non-rigid shape registration method is proposed,called balanced functional maps(BFM).The BFM algorithm generalizes the point-based correspondence to functions.By choosing the Laplace-Beltrami eigenfunctions as the function basis,the transformations between shapes can be represented by the functional map(FM)matrix.In addition,many constraints on shape registration,such as the feature descriptor,keypoint,and salient region correspondence,can be formulated linearly using the matrix.By bi-directionally searching for the nearest neighbors of points’indicator functions in the function space,the point-based correspondence can be derived from FMs.We conducted several experiments on the Topology and Orchestration Specification for Cloud Applications(TOSCA)dataset and the Shape Completion and Animation of People(SCAPE)dataset.Experimental results show that the proposed BFM algorithm is effective and has superior performance than the state-of-the-art methods on both datasets.

3D Shape Reconstruction of Lumbar Vertebra From Two X-ray Images and a CT Model

Structure reconstruction of 3 D anatomy from biplanar X-ray images is a challenging topic. Traditionally, the elastic-model-based method was used to reconstruct 3 D shapes by deforming the control points on the elastic mesh. However, the reconstructed shape is not smooth because the limited control points are only distributed on the edge of the elastic mesh.Alternatively, statistical-model-based methods, which include shape-model-based and intensity-model-based methods, are introduced due to their smooth reconstruction. However, both suffer from limitations. With the shape-model-based method, only the boundary profile is considered, leading to the loss of valid intensity information. For the intensity-based-method, the computation speed is slow because it needs to calculate the intensity distribution in each iteration. To address these issues, we propose a new reconstruction method using X-ray images and a specimen’s CT data. Specifically, the CT data provides both the shape mesh and the intensity model of the vertebra. Intensity model is used to generate the deformation field from X-ray images, while the shape model is used to generate the patient specific model by applying the calculated deformation field.Experiments on the public synthetic dataset and clinical dataset show that the average reconstruction errors are 1.1 mm and1.2 mm, separately. The average reconstruction time is 3 minutes.

Rui Liao's work on patient-specific 3-D model guidance for interventional and hybrid-operating-room applications

Compared to surgery,interventional and hybrid-operating-room(OR) approaches diagnose or treat pathology with the most minimally invasive techniques possible.By minimizing the physical trauma to the patient,peripheral or hybrid approaches can reduce infection rates and recovery time as well as shorten hospital stays.Minimally invasive approaches therefore are the trend and often the preferred choice,and may even be the only option for the patients associated with high surgery risks.Common interventional imaging modalities include 2-D X-ray fluoroscopy and ultrasound.However,fluoroscopic images do not display the anatomic structures without a contrast agent,which on the other hand,needs to be minimized for patients' safety.Ultrasound images suffer from relatively low image quality and tissue contrast problems.To augment the doctor's view of the patient's anatomy and help doctors navigate the devices to the targeted area with more confidence and a higher accuracy,high-resolution pre-operative volumetric data such as computed tomography and/or magnetic resonance can be fused with intra-operative 2-D images during interventions.A seamless workflow and accurate 2-D/3-D registrationas well as cardiac and/or respiratory motion compensation are the key components for a successful image guidance system using a patient-specific 3-D model.Dr.Liao's research has been focused on developing methods and systems of 3-D model guidance for various interventions and hybrid-OR applications.Dr.Liao' s work has led to several Siemens products with high clinical and/or market impact and a good number of scientific publications in leading journals/conferences on medical imaging.

RANGE ACQUISITION IN BINOCULAR 3-D COMPUTER VISION USING EDGE-BASED HIERARCHICAL MATCHING

This paper deals with a binocular 3-D computer vision system based on the hierarchicalmatching of edge features, Frei and Chen operator is used to extract the edge. The averagegradients of an image obtained by two isotropic operators are non-equal quantized andthresholded in an angle, Edge features are extracted after passing a preemphasis transferfunction which can equalize, the noise affection. Binary edge images are decomposed into apyramid structure which is stored and searched using llliffe’s location method. Corre-sponding points are used to determine the range data using triangulation based on an improvedTrivedi’s formula. In calibration the authors set the optical axes of the two cameras parallelto simplify the calculation, A 3 rd order Householder transform is used to solve the compati-ble coupled equations.

A study of projections for key point based registration of panoramic terrestrial 3D laser scan

This paper surveys state-of-the-art image features and descriptors for the task of 3D scan registration based on panoramic reflectance images.As modern terrestrial laser scanners digitize their environment in a spherical way,the sphere has to be projected to a two-dimensional image.To this end,we evaluate the equirectangular,the cylindrical,the Mercator,the rectilinear,the Pannini,the stereographic,and the z-axis projection.We show that the Mercator and the Pannini projection outperform the other projection methods.

A survey on image and video stitching

Image/video stitching is a technology for solving the field of view(FOV)limitation of images/videos.It stitches multiple overlapping images/videos to generate a wide-FOV image/video,and has been used in various fields such as sports broadcasting,video surveillance,street view,and entertainment.This survey reviews image/video stitching algorithms,with a particular focus on those developed in recent years.Image stitching first calculates the corresponding relationships between multiple overlapping images,deforms and aligns the matched images,and then blends the aligned images to generate a wide-FOV image.A seamless method is always adopted to eliminate such potential flaws as ghosting and blurring caused by parallax or objects moving across the overlapping regions.Video stitching is the further extension of image stitching.It usually stitches selected frames of original videos to generate a stitching template by performing image stitching algorithms,and the subsequent frames can then be stitched according to the template.Video stitching is more complicated with moving objects or violent camera movement,because these factors introduce jitter,shakiness,ghosting,and blurring.Foreground detection technique is usually combined into stitching to eliminate ghosting and blurring,while video stabilization algorithms are adopted to solve the jitter and shakiness.This paper further discusses panoramic stitching as a special-extension of image/video stitching.Panoramic stitching is currently the most widely used application in stitching.This survey reviews the latest image/video stitching methods,and introduces the fundamental principles/advantages/weaknesses of image/video stitching algorithms.Image/video stitching faces long-term challenges such as wide baseline,large parallax,and low-texture problem in the overlapping region.New technologies may present new opportunities to address these issues,such as deep learning-based semantic correspondence,and 3D image stitching.Finally,this survey discusses the challenges of image/video stitching and proposes potential solutions.

Skyline-Based Registration of 3D Laser Scans

Acquisition and registration of terrestrial 3D laser scans is a fundamental task in mapping and modeling of cities in three dimensions. To automate this task marker-flee registration methods are required. Based on the existence of skyline features, this paper proposes a novel method. The skyline features are extracted from panoramic 3D scans and encoded as strings enabling the use of string matching for merging the scans. Initial results of the proposed method in the old city center of Bremen are presented.

A diffusion-weighted imaging based diagnostic system for early detection of prostate cancer

A new framework for early diagnosis of prostate cancer using Diffusion-Weighted Imaging (DWI) is proposed. The proposed diagnostic approach consists of the following four steps to detect locations that are suspicious for prostate cancer: 1) In the first step, we isolate the prostate from the surrounding anatomical structures based on a Maximum A Posteriori (MAP) estimate of a new log-likelihood function that accounts for the shape priori, the spatial interaction, and the current appearance of prostate tissues and its background (surrounding anatomical structures);2) In order to take into account any local deformation between the segmented prostates at different b-values that could occur during the scanning process due to local motion, a non-rigid registration algorithm is employed;3) A KNN-based classifier is used to classify the prostate into benign or malignant based on three appearance features extracted from registered images;and 4) The tumor boundaries are determined using a level set deformable model controlled by the diffusion information and the spatial interactions between the prostate voxels. Preliminary experiments on 28 patients (17 malignant and 11 benign) resulted in 100% correct classification, showing that the proposed method is a promising supplement to current technologies (biopsy-based diagnostic systems) for the early diagnosis of prostate cancer.

3D face recognition based on principal axes registration and fusing features

A 3D face recognition approach which uses principal axes registration(PAR)and three face representation features from the re-sampling depth image:Eigenfaces,Fisherfaces and Zernike moments is presented.The approach addresses the issue of 3D face registration instantly achieved by PAR.Because each facial feature has its own advantages,limitations and scope of use,different features will complement each other.Thus the fusing features can learn more expressive characterizations than a single feature.The support vector machine(SVM)is applied for classification.In this method,based on the complementarity between different features,weighted decision-level fusion makes the recognition system have certain fault tolerance.Experimental results show that the proposed approach achieves superior performance with the rank-1 recognition rate of 98.36%for GavabDB database.

PuzzleNet:Boundary-Aware Feature Matching for Non-Overlapping 3D Point Clouds Assembly

We address the 3D shape assembly of multiple geometric pieces without overlaps, a scenario often encountered in 3D shape design, field archeology, and robotics. Existing methods depend on strong assumptions on the number of shape pieces and coherent geometry or semantics of shape pieces. Despite raising attention to 3D registration with complex or low overlapping patterns, few methods consider shape assembly with rare overlaps. To address this problem, we present a novel framework inspired by solving puzzles, named PuzzleNet, which conducts multi-task learning by leveraging both 3D alignment and boundary information. Specifically, we design an end-to-end neural network based on a point cloud transformer with two-way branches for estimating rigid transformation and predicting boundaries simultaneously. The framework is then naturally extended to reassemble multiple pieces into a full shape by using an iterative greedy approach based on the distance between each pair of candidate-matched pieces. To train and evaluate PuzzleNet, we construct two datasets, named ModelPuzzle and DublinPuzzle, based on a real-world urban scan dataset (DublinCity) and a synthetic CAD dataset (ModelNet40) respectively. Experiments demonstrate our effectiveness in solving 3D shape assembly for multiple pieces with arbitrary geometry and inconsistent semantics. Our method surpasses state-of-the-art algorithms by more than 10 times in rotation metrics and four times in translation metrics.

Local voxelizer:A shape descriptor for surface registration

Surface registration brings multiple scans into a common coordinate system by aligning their overlapping components. This can be achieved by finding a few pairs of matched points on different scans using local shape descriptors and employing the matches to compute transformations to produce the alignment. By defining a unique local reference frame(LRF) and attaching an LRF to shape descriptors,the transformation can be computed using only one match based on aligning the LRFs. This paper proposes a local voxelizer descriptor,and the key ideas are to define a unique LRF using the support around a basis point,to perform voxelization for the local shape within a cubical volume aligned with the LRF,and to concatenate local features extracted from each voxel to construct the descriptor. An automatic rigid registration approach is given based on the local voxelizer and an expanding strategy that merges descriptor representations of aligned scans. Experiments show that our registration approach allows the acquisition of 3D models of various objects,and that the local voxelizer is robust to mesh noise and varying mesh resolution,in comparison to two state-of-the-art shape descriptors.

Improved Spatial Registration Algorithm for Sensors on Multiple Mobile Platforms

This paper focuses on the spatial registration algorithm under the earth-center earth-fixed(ECEF)coordinate system for multiple mobile platforms.The sensor measurement biases are discussed with the attitude information of the platform into consideration.First,the biased measurement model is constructed.Besides,the maximum likelihood registration(MLR)algorithm is discussed to simultaneously estimate the measurement biases and the target state.Finally,an improved online MLR(IMLR)algorithm is proposed through a sliding window of adaptive size.Simulation results demonstrate that the proposed IMLR algorithm effectively improves the realtime ability of the system and can approach similar estimation accuracy to the conventional MLR algorithm.

地面三维激光扫描的点云配准误差研究

针对闭合条件下地面三维激光扫描点云配准产生的闭合差,基于测量平差理论,提出一种闭合差分配方法。首先通过间接平差理论求出各相邻测站间的坐标转换参数及其精度,再对闭合差按与方差成正比分配给各测站。通过多站点云配准实验,验证了该方法能在一定程度上提高点云配准的精度。