GAN-DIRNet:A Novel Deformable Image Registration Approach for Multimodal Histological Images

Multi-modal histological image registration tasks pose significant challenges due to tissue staining operations causing partial loss and folding of tissue.Convolutional neural network(CNN)and generative adversarial network(GAN)are pivotal inmedical image registration.However,existing methods often struggle with severe interference and deformation,as seen in histological images of conditions like Cushing’s disease.We argue that the failure of current approaches lies in underutilizing the feature extraction capability of the discriminator inGAN.In this study,we propose a novel multi-modal registration approach GAN-DIRNet based on GAN for deformable histological image registration.To begin with,the discriminators of two GANs are embedded as a new dual parallel feature extraction module into the unsupervised registration networks,characterized by implicitly extracting feature descriptors of specific modalities.Additionally,modal feature description layers and registration layers collaborate in unsupervised optimization,facilitating faster convergence and more precise results.Lastly,experiments and evaluations were conducted on the registration of the Mixed National Institute of Standards and Technology database(MNIST),eight publicly available datasets of histological sections and the Clustering-Registration-Classification-Segmentation(CRCS)dataset on the Cushing’s disease.Experimental results demonstrate that our proposed GAN-DIRNet method surpasses existing approaches like DIRNet in terms of both registration accuracy and time efficiency,while also exhibiting robustness across different image types.

Free form deformation and symmetry constraint‐based multimodal brain image registration using generative adversarial nets

Multi‐modal brain image registration has been widely applied to functional localisation,neurosurgery and computational anatomy.The existing registration methods based on the dense deformation fields involve too many parameters,which is not conducive to the exploration of correct spatial correspondence between the float and reference images.Meanwhile,the unidirectional registration may involve the deformation folding,which will result in the change of topology during registration.To address these issues,this work has presented an unsupervised image registration method using the free form deformation(FFD)and the symmetry constraint‐based generative adversarial networks(FSGAN).The FSGAN utilises the principle component analysis network‐based structural representations of the reference and float images as the inputs and uses the generator to learn the FFD model parameters,thereby producing two deformation fields.Meanwhile,the FSGAN uses two discriminators to decide whether the bilateral registration have been realised simultaneously.Besides,the symmetry constraint is utilised to construct the loss function,thereby avoiding the deformation folding.Experiments on BrainWeb,high grade gliomas,IXI and LPBA40 show that compared with state‐of‐the‐art methods,the FSGAN provides superior performance in terms of visual comparisons and such quantitative indexes as dice value,target registration error and computational efficiency.

Application of Opening and Closing Morphology in Deep Learning-Based Brain Image Registration

In order to improve the registration accuracy of brain magnetic resonance images(MRI),some deep learning registration methods use segmentation images for training model.How-ever,the segmentation values are constant for each label,which leads to the gradient variation con-centrating on the boundary.Thus,the dense deformation field(DDF)is gathered on the boundary and there even appears folding phenomenon.In order to fully leverage the label information,the morphological opening and closing information maps are introduced to enlarge the non-zero gradi-ent regions and improve the accuracy of DDF estimation.The opening information maps supervise the registration model to focus on smaller,narrow brain regions.The closing information maps supervise the registration model to pay more attention to the complex boundary region.Then,opening and closing morphology networks(OC_Net)are designed to automatically generate open-ing and closing information maps to realize the end-to-end training process.Finally,a new registra-tion architecture,VM_(seg+oc),is proposed by combining OC_Net and VoxelMorph.Experimental results show that the registration accuracy of VM_(seg+oc) is significantly improved on LPBA40 and OASIS1 datasets.Especially,VM_(seg+oc) can well improve registration accuracy in smaller brain regions and narrow regions.

Multi-modality liver image registration based on multilevel B-splines free-form deformation and L-BFGS optimal algorithm

A new coarse-to-fine strategy was proposed for nonrigid registration of computed tomography(CT) and magnetic resonance(MR) images of a liver.This hierarchical framework consisted of an affine transformation and a B-splines free-form deformation(FFD).The affine transformation performed a rough registration targeting the mismatch between the CT and MR images.The B-splines FFD transformation performed a finer registration by correcting local motion deformation.In the registration algorithm,the normalized mutual information(NMI) was used as similarity measure,and the limited memory Broyden-Fletcher- Goldfarb-Shannon(L-BFGS) optimization method was applied for optimization process.The algorithm was applied to the fully automated registration of liver CT and MR images in three subjects.The results demonstrate that the proposed method not only significantly improves the registration accuracy but also reduces the running time,which is effective and efficient for nonrigid registration.

A hybrid genetic algorithm for multi-modal image registration

This paper describes a new method for three-dimensional medical image registration. In the interactive image-guided HIFU （ High Intensity Focused Ultrasound） therapy system, a fast and precise localization of the tumor is very important. An automatic system is developed for registering pre-operative MR images with intra-operative ultrasound images based on the vessels visible in both of the modalities. When the MR and the ultrasound images are aligned, the eenterline points of the vessels in the MR image will align with bright intensities in the ultrasound image. The method applies an optimization strategy combining the genetic algorithm with the conjugated gradients algorithm to minimize the objective function. It provides a feasible way of determining the global solution and makes the method robust to local maximum and insensitive to initial position. Two experiments were designed to evaluate the method, and the results show that our method has better registration accuracy and convergence rate than the other two classic algorithms.

Robust Image Registration Based on Mutual Information Measure

A new implementation of the image registration algorithm based on the mutual information is presented for the case of medical images. The registration is achieved if the maximum of the mutual information is attained. In this maximization process optimal values of five parameters of an affine transformation are searched.

A method based on mutual information and gradient information for medical image registration

Mutual information is widely used in medical image registration, because it does not require preprocessing the image. However, the local maximum problem in the registration is insurmountable. We combine mutual information and gradient information to solve this problem and apply it to the non-rigid deformation image registration. To improve the accuracy, we provide some implemental issues, for example, the Powell searching algorithm, gray interpolation and consideration of outlier points. The experimental results show the accuracy of the method and the feasibility in non-rigid medical image registration.

A Review of Point Feature Based Medical Image Registration

Point features, as the basis of lines, surfaces, and bodies, are commonly used in medical image registration. To obtain an elegant spatial transformation of extracted feature points, many point set matching algorithms(PMs) have been developed to match two point sets by optimizing multifarious distance functions. There are ample reviews related to medical image registration and PMs which summarize their basic principles and main algorithms separately. However,to data, detailed summary of PMs used in medical image registration in different clinical environments has not been published. In this paper, we provide a comprehensive review of the existing key techniques of the PMs applied to medical image registration according to the basic principles and clinical applications. As the core technique of the PMs, geometric transformation models are elaborated in this paper, demonstrating the mechanism of point set registration. We also focus on the clinical applications of the PMs and propose a practical classification method according to their applications in different clinical surgeries. The aim of this paper is to provide a summary of pointfeaturebased methods used in medical image registration and to guide doctors or researchers interested in this field to choose appropriate techniques in their research.

SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness

Image fusion aims to integrate complementary information in source images to synthesize a fused image comprehensively characterizing the imaging scene. However, existing image fusion algorithms are only applicable to strictly aligned source images and cause severe artifacts in the fusion results when input images have slight shifts or deformations. In addition,the fusion results typically only have good visual effect, but neglect the semantic requirements of high-level vision tasks.This study incorporates image registration, image fusion, and semantic requirements of high-level vision tasks into a single framework and proposes a novel image registration and fusion method, named Super Fusion. Specifically, we design a registration network to estimate bidirectional deformation fields to rectify geometric distortions of input images under the supervision of both photometric and end-point constraints. The registration and fusion are combined in a symmetric scheme, in which while mutual promotion can be achieved by optimizing the naive fusion loss, it is further enhanced by the mono-modal consistent constraint on symmetric fusion outputs. In addition, the image fusion network is equipped with the global spatial attention mechanism to achieve adaptive feature integration. Moreover, the semantic constraint based on the pre-trained segmentation model and Lovasz-Softmax loss is deployed to guide the fusion network to focus more on the semantic requirements of high-level vision tasks. Extensive experiments on image registration, image fusion,and semantic segmentation tasks demonstrate the superiority of our Super Fusion compared to the state-of-the-art alternatives.The source code and pre-trained model are publicly available at https://github.com/Linfeng-Tang/Super Fusion.

Total Variation Constrained Non-Negative Matrix Factorization for Medical Image Registration

This paper presents a novel medical image registration algorithm named total variation constrained graphregularization for non-negative matrix factorization(TV-GNMF).The method utilizes non-negative matrix factorization by total variation constraint and graph regularization.The main contributions of our work are the following.First,total variation is incorporated into NMF to control the diffusion speed.The purpose is to denoise in smooth regions and preserve features or details of the data in edge regions by using a diffusion coefficient based on gradient information.Second,we add graph regularization into NMF to reveal intrinsic geometry and structure information of features to enhance the discrimination power.Third,the multiplicative update rules and proof of convergence of the TV-GNMF algorithm are given.Experiments conducted on datasets show that the proposed TV-GNMF method outperforms other state-of-the-art algorithms.

A NEW IMAGE REGISTRATION METHOD FOR GREY IMAGES

The proposed algorithm relies on a group of new formulas for calculating tangent slope so as to address angle feature of edge curves of image. It can utilize tangent angle features to estimate automatically and fully the rotation parameters of geometric transform and enable rough matching of images with huge rotation difference. After angle compensation, it can search for matching point sets by correlation criterion, then calculate parameters of affine transform, enable higher-precision emendation of rotation and transferring. Finally, it fulfills precise matching for images with relax-tense iteration method. Compared with the registration approach based on wavelet direction-angle features, the matching algorithm with tangent feature of image edge is more robust and realizes precise registration of various images. Furthermore, it is also helpful in graphics matching.

Multi-sensor image registration using multi-resolution shape analysis

Multi-sensor image registration has been widely used in remote sensing and medical image field, but registration performance is degenerated when heterogeneous images are involved. An image registration method based on multi-resolution shape analysis is proposed in this paper, to deal with the problem that the shape of similar objects is always invariant. The contours of shapes are first detected as visual features using an extended contour search algorithm in order to reduce effects of noise, and the multi-resolution shape descriptor is constructed through Fourier curvature representation of the contour’s chain code. Then a minimum distance function is used to judge the similarity between two contours. To avoid the effect of different resolution and intensity distribution, suitable resolution of each image is selected by maximizing the consistency of its pyramid shapes. Finally, the transformation parameters are estimated based on the matched control-point pairs which are the centers of gravity of the closed contours. Multi-sensor Landsat TM imagery and infrared imagery have been used as experimental data for comparison with the classical contour-based registration. Our results have been shown to be superior to the classical ones.

Optical and SAR image registration based on improved nonsubsampled wavelet transform for sea islands

Homologous feature point extraction is a key problem in the optical and synthetic aperture radar （SAR） image registration for islands. A new feature point extraction method using a threshold shrink operator and non-subsampled wavelet transform （TSO-NSWT） for optical and SAR image registration was proposed. Moreover, the matching for this proposed feature was different from the traditional feature matching strategies and was performed using a similarity measure computed from neighborhood circles in low-frequency bands. Then, a number of reliably matched couples with even distributions were obtained, which assured the accuracy of the registration. Application of the proposed algorithm to SPOT-5 （multi-spectral） and YG-1 （SAR） images showed that a large number of accurately matched couples could be identified. Additionally, both of the root mean square error （RMSE） patterns of the registration parameters computed based on the TSO-NSWT algorithm and traditional NSWT algorithm were analyzed and compared, which further demonstrated the effectiveness of the proposed algorithm. The algorithm can supply the crucial step for island image registration and island recognition.

Two fully automated data-driven 3D whole-breast segmentation strategies in MRI for MR-based breast density using image registration and U-Net with a focus on reproducibility

Presence of higher breast density(BD)and persistence over time are risk factors for breast cancer.A quantitatively accurate and highly reproducible BD measure that relies on precise and reproducible whole-breast segmentation is desirable.In this study,we aimed to develop a highly reproducible and accurate whole-breast segmentation algorithm for the generation of reproducible BD measures.Three datasets of volunteers from two clinical trials were included.Breast MR images were acquired on 3T Siemens Biograph mMR,Prisma,and Skyra using 3D Cartesian six-echo GRE sequences with a fat-water separation technique.Two whole-breast segmentation strategies,utiliz-ing image registration and 3D U-Net,were developed.Manual segmentation was performed.A task-based analysis was performed:a previously developed MR-based BD measure,MagDensity,was calculated and assessed using automated and manual segmentation.The mean squared error(MSE)and intraclass correlation coefficient(ICC)between MagDensity were evaluated using the manual segmentation as a reference.The test-retest reproducibility of MagDensity derived from different breast segmentation methods was assessed using the difference between the test and retest measures(Δ_(2-1)),MSE,and ICC.The results showed that MagDensity derived by the registration and deep learning segmentation methods exhibited high concordance with manual segmentation,with ICCs of 0.986(95%CI:0.974-0.993)and 0.983(95%CI:0.961-0.992),respectively.For test-retest analysis,MagDensity derived using the regis-tration algorithm achieved the smallest MSE of 0.370 and highest ICC of 0.993(95%CI:0.982-0.997)when compared to other segmentation methods.In conclusion,the proposed registration and deep learning whole-breast segmentation methods are accurate and reliable for estimating BD.Both methods outperformed a previously developed algorithm and manual segmentation in the test-retest assessment,with the registration exhibiting superior performance for highly reproducible BD measurements.

Sonar Image Registration and Mosaic Based on Line Detection and Triangle Matching

Image registration is an important research topic in the field of computer vision,in which the registration and mosaic of side-scan sonar images is the keypoints of underwater navigation.However,the image registration method of keypoints is not suitable for sonar images which do not have obvious feature points.Therefore,a method of sonar-image registration and mosaic based on line segment extraction and triangle matching is proposed in this paper.Firstly,in order to extract features from sonar image,the LSD method is introduced to detect line feature from images,and line segments are filtered by the principle of attention;after that,triangles are formed from line segments,an image transformation matrix can be calculated through the heuristic greedy algorithm from these triangles;finally,images are merged based on the transformation information.On the basis of practical tests,it is found that,the feature extraction method used in this paper can better describe the outline of underwater terrain,and there is no obvious stitching gap between the result of sonar images stitched.Experimental results show that the proposed method is effective than the keypoints method of the registration and mosaic of sonar images.

Image Registration Based on Improved Mutual Information with Hybrid Optimizer

An improved image registration method is proposed based on mutual infor- mation with hybrid optimizer. Firstly, mutual information measure is combined with morphological gradient information. The essence of the gradient information is that locations a large gradient magnitude should be aligned, but also the orientation of the gradients at those locations should be similar. Secondly, a hybrid optimizer combined PSO with Powell algorithm is proposed to restrain local maxima of mutual information function and improve the registration accuracy to sub-pixel level. Lastly, muhlresolution data structure based on Mallat decomposition can not only improve the behavior of registration function, but also improve the speed of the algorithm. Experimental results demonstrate that the new method can yield good registration result, superior to traditional optimizer with respect to smoothness and attraction basin as well as convergence speed.

Two new approaches for image registration based onspatial-temporal relationship

How to improve the probability of registration and precision of localization is a hard problem, which is desiderated to solve. The two basic approaches (normalized cross-correlation and phase correlation) for image registration are analysed, two improved approaches based on spatial-temporal relationship are presented. This method adds the correlation matrix according to the displacements in x- cirection and y- directions, and the registration pose is searched in the added matrix. The method overcomes the shortcoming that the probability of registration decreasing with area increasing owing to geometric distortion, improves the probability and the robustness of registration.

Medical Image Registration Based on Phase Congruency and Regional Mutual Information

In this paper, a new approach of muhi-modality image registration is represented with not only image intensity, but also features describing image structure. There are two novelties in the proposed method. Firstly, instead of standard mutual information （ MI ） based on joint intensity histogram, regional mutual information （ RMI ） is employed, which allows neighborhood information to be taken into account. Secondly, a new feature images obtained by means of phase congruency are invariants to brightness or contrast changes. By incorporating these features and intensity into RMI, we can combine the aspects of both structural and neighborhood information together, which offers a more robust and a high level of registration accuracy.

Automatic Image Registration Algorithm Based on Wavelet Transform

An automatic image registration approach based on wavelet transform is proposed. This proposed method utilizes multiscale wavelet transform to extract feature points. A coarse-to-fine feature matching method is utilized in the feature matching phase. A two-way matching method based on cross-correlation to get candidate point pairs and a fine matching based on support strength combine to form the matching algorithm. At last, based on an affine transformation model, the parameters are iteratively refined by using the leastsquares estimation approach. Experimental results have verified that the proposed algorithm can realize automatic registration of various kinds of images rapidly and effectively.

Constrained branch-and-bound algorithm for image registration

In this paper, the authors propose a refined Branch-and-Bound algorithm for affine-transformation based image registration. Given two feature point-sets in two images respectively, the authors first extract a sequence of high-probability matched point-pairs by considering well-defined features. Each resultant point-pair can be regarded as a constraint in the search space of Branch-and-Bound algorithm guiding the search process. The authors carry out Branch-and-Bound search with the constraint of a pair-point selected by using Monte Carlo sampling according to the match measures of point-pairs. If such one cannot lead to correct result, additional candidate is chosen to start another search. High-probability matched point-pairs usually results in fewer loops and the search process is accelerated greatly. Experimental results verify the high efficiency and robustness of the author’s approach.