GAN-DIRNet:A Novel Deformable Image Registration Approach for Multimodal Histological Images

Multi-modal histological image registration tasks pose significant challenges due to tissue staining operations causing partial loss and folding of tissue.Convolutional neural network(CNN)and generative adversarial network(GAN)are pivotal inmedical image registration.However,existing methods often struggle with severe interference and deformation,as seen in histological images of conditions like Cushing’s disease.We argue that the failure of current approaches lies in underutilizing the feature extraction capability of the discriminator inGAN.In this study,we propose a novel multi-modal registration approach GAN-DIRNet based on GAN for deformable histological image registration.To begin with,the discriminators of two GANs are embedded as a new dual parallel feature extraction module into the unsupervised registration networks,characterized by implicitly extracting feature descriptors of specific modalities.Additionally,modal feature description layers and registration layers collaborate in unsupervised optimization,facilitating faster convergence and more precise results.Lastly,experiments and evaluations were conducted on the registration of the Mixed National Institute of Standards and Technology database(MNIST),eight publicly available datasets of histological sections and the Clustering-Registration-Classification-Segmentation(CRCS)dataset on the Cushing’s disease.Experimental results demonstrate that our proposed GAN-DIRNet method surpasses existing approaches like DIRNet in terms of both registration accuracy and time efficiency,while also exhibiting robustness across different image types.

Nonlinear Registration of Brain Magnetic Resonance Images with Cross Constraints of Intensity and Structure

Many deep learning-based registration methods rely on a single-stream encoder-decoder network for computing deformation fields between 3D volumes.However,these methods often lack constraint information and overlook semantic consistency,limiting their performance.To address these issues,we present a novel approach for medical image registration called theDual-VoxelMorph,featuring a dual-channel cross-constraint network.This innovative network utilizes both intensity and segmentation images,which share identical semantic information and feature representations.Two encoder-decoder structures calculate deformation fields for intensity and segmentation images,as generated by the dual-channel cross-constraint network.This design facilitates bidirectional communication between grayscale and segmentation information,enabling the model to better learn the corresponding grayscale and segmentation details of the same anatomical structures.To ensure semantic and directional consistency,we introduce constraints and apply the cosine similarity function to enhance semantic consistency.Evaluation on four public datasets demonstrates superior performance compared to the baselinemethod,achieving Dice scores of 79.9%,64.5%,69.9%,and 63.5%for OASIS-1,OASIS-3,LPBA40,and ADNI,respectively.

Real-time continuous image guidance for endoscopic retrograde cholangiopancreatography based on 3D/2D registration and respiratory compensation

BACKGROUND It has been confirmed that three-dimensional(3D)imaging allows easier identification of bile duct anatomy and intraoperative guidance of endoscopic retrograde cholangiopancreatography(ERCP),which reduces the radiation dose and procedure time with improved safety.However,current 3D biliary imaging does not have good real-time fusion with intraoperative imaging,a process meant to overcome the influence of intraoperative respiratory motion and guide navigation.The present study explored the feasibility of real-time continuous image-guided ERCP.AIM To explore the feasibility of real-time continuous image-guided ERCP.METHODS We selected 23D-printed abdominal biliary tract models with different structures to simulate different patients.The ERCP environment was simulated for the biliary phantom experiment to create a navigation system,which was further tested in patients.In addition,based on the estimation of the patient’s respiratory motion,preoperative 3D biliary imaging from computed tomography of 18 patients with cholelithiasis was registered and fused in real-time with 2D fluoroscopic sequence generated by the C-arm unit during ERCP.RESULTS Continuous image-guided ERCP was applied in the biliary phantom with a registration error of 0.46 mm±0.13 mm and a tracking error of 0.64 mm±0.24mm.After estimating the respiratory motion,3D/2D registration accurately transformed preoperative 3D biliary images to each image in the X-ray image sequence in real-time in 18 patients,with an average fusion rate of 88%.CONCLUSION Continuous image-guided ERCP may be an effective approach to assist the operator and reduce the use of X-ray and contrast agents.

Free form deformation and symmetry constraint‐based multimodal brain image registration using generative adversarial nets

Multi‐modal brain image registration has been widely applied to functional localisation,neurosurgery and computational anatomy.The existing registration methods based on the dense deformation fields involve too many parameters,which is not conducive to the exploration of correct spatial correspondence between the float and reference images.Meanwhile,the unidirectional registration may involve the deformation folding,which will result in the change of topology during registration.To address these issues,this work has presented an unsupervised image registration method using the free form deformation(FFD)and the symmetry constraint‐based generative adversarial networks(FSGAN).The FSGAN utilises the principle component analysis network‐based structural representations of the reference and float images as the inputs and uses the generator to learn the FFD model parameters,thereby producing two deformation fields.Meanwhile,the FSGAN uses two discriminators to decide whether the bilateral registration have been realised simultaneously.Besides,the symmetry constraint is utilised to construct the loss function,thereby avoiding the deformation folding.Experiments on BrainWeb,high grade gliomas,IXI and LPBA40 show that compared with state‐of‐the‐art methods,the FSGAN provides superior performance in terms of visual comparisons and such quantitative indexes as dice value,target registration error and computational efficiency.

Application of Opening and Closing Morphology in Deep Learning-Based Brain Image Registration

In order to improve the registration accuracy of brain magnetic resonance images(MRI),some deep learning registration methods use segmentation images for training model.How-ever,the segmentation values are constant for each label,which leads to the gradient variation con-centrating on the boundary.Thus,the dense deformation field(DDF)is gathered on the boundary and there even appears folding phenomenon.In order to fully leverage the label information,the morphological opening and closing information maps are introduced to enlarge the non-zero gradi-ent regions and improve the accuracy of DDF estimation.The opening information maps supervise the registration model to focus on smaller,narrow brain regions.The closing information maps supervise the registration model to pay more attention to the complex boundary region.Then,opening and closing morphology networks(OC_Net)are designed to automatically generate open-ing and closing information maps to realize the end-to-end training process.Finally,a new registra-tion architecture,VM_(seg+oc),is proposed by combining OC_Net and VoxelMorph.Experimental results show that the registration accuracy of VM_(seg+oc) is significantly improved on LPBA40 and OASIS1 datasets.Especially,VM_(seg+oc) can well improve registration accuracy in smaller brain regions and narrow regions.

Automated Registration for Infrared Image Based on Wavelet Analysis

To develop a quick, accurate and antinoise automated image registration technique for infrared images, the wavelet analysis technique was used to extract the feature points in two images followed by the compensation for input image with angle difference between them. A hi erarchical feature matching algorithm was adopted to get the final transform parameters between the two images. The simulation results for two infrared images show that the method can effectively, quickly and accurately register images and be antinoise to some extent.

Projection registration of X-ray image and CT image

A methodology for alignment of an X-ray image and a CT image, based on the Chamfer 3-4 distance transform and simulated annealing optimization algorithm is presented. Firstly, an initial transformation matrix is constructed. For the convenience of computing, geometric models of the X-ray device to reconstruct the calibration matrix are used. Then, by defining the distance between the 3-D protective and the 2-D object image, we optimize this distance matching problem, using the simulated annealing algorithm. This method is also integrated into medical intra-operation, dealing with the data set acquired from 3-D image workstation and active navigation.

IMAGE PANORAMIC MOS AICING WITH GLOBAL AND LOCAL REGISTRATION

Technique s for constructing full view panoramic mosaics from sequences of images are pres ented. The goal of this work is to remove too many limitations for pure panning motion. The best reference block is important for the block-matching method for improving the robustness and performance. It is automatically selected in the h igh-frequency image, which always contains the plenty visible features. In orde r to reduce accumulated registration errors, the global registration using the p hase-correlation matching method with rotation adjustment is applied to the who le sequence of images, which results in an optimal image mosaic with resolving t ranslational or rotational motion. The local registration using the Levenberg-M arquardt iterative non-linear minimization algorithm is applied to compensating for small amounts of motion parallax introduced by translations of the camera a nd other unmodeled distortions, then minimizing the discrepancy after applying t he global registration. The accumulated misregistration errors may cause a visib le gap between the two images. A smoothing filter is introduced for removing the visible artifact.

Novel registration algorithm for 3-D images captured from multiple views of object surface

A novel algorithm of 3-D surface image registration is proposed. It makes use of the array information of 3-D points and takes vector/vertex-like features as the basis of the matching. That array information of 3-D points can be easily obtained when capturing original 3-D images. The iterative least-mean-squared （LMS） algorithm is applied to optimizing adaptively the transformation matrix parameters. These can effectively improve the registration performance and hurry up the matching process. Experimental results show that it can reach a good subjective impression on aligned 3-D images. Although the algorithm focuses primarily on the human head model, it can also be used for other objects with small modifications.

Research on Detection Technology of Micro-Components on Circuit Board Based on Digital Image Processing

Aiming at the stability of the circuit board image in the acquisition process,this paper realizes the accurate registration of the image to be registered and the standard image based on the SIFT feature operator and RANSAC algorithm.The device detection model and data set are established based on Faster RCNN.Finally,the number of training was continuously optimized,and when the loss function of Faster RCNN converged,the identification result of the device was obtained.

A method based on mutual information and gradient information for medical image registration

Mutual information is widely used in medical image registration, because it does not require preprocessing the image. However, the local maximum problem in the registration is insurmountable. We combine mutual information and gradient information to solve this problem and apply it to the non-rigid deformation image registration. To improve the accuracy, we provide some implemental issues, for example, the Powell searching algorithm, gray interpolation and consideration of outlier points. The experimental results show the accuracy of the method and the feasibility in non-rigid medical image registration.

A Review of Point Feature Based Medical Image Registration

Point features, as the basis of lines, surfaces, and bodies, are commonly used in medical image registration. To obtain an elegant spatial transformation of extracted feature points, many point set matching algorithms(PMs) have been developed to match two point sets by optimizing multifarious distance functions. There are ample reviews related to medical image registration and PMs which summarize their basic principles and main algorithms separately. However,to data, detailed summary of PMs used in medical image registration in different clinical environments has not been published. In this paper, we provide a comprehensive review of the existing key techniques of the PMs applied to medical image registration according to the basic principles and clinical applications. As the core technique of the PMs, geometric transformation models are elaborated in this paper, demonstrating the mechanism of point set registration. We also focus on the clinical applications of the PMs and propose a practical classification method according to their applications in different clinical surgeries. The aim of this paper is to provide a summary of pointfeaturebased methods used in medical image registration and to guide doctors or researchers interested in this field to choose appropriate techniques in their research.

SuperFusion: A Versatile Image Registration and Fusion Network with Semantic Awareness

Image fusion aims to integrate complementary information in source images to synthesize a fused image comprehensively characterizing the imaging scene. However, existing image fusion algorithms are only applicable to strictly aligned source images and cause severe artifacts in the fusion results when input images have slight shifts or deformations. In addition,the fusion results typically only have good visual effect, but neglect the semantic requirements of high-level vision tasks.This study incorporates image registration, image fusion, and semantic requirements of high-level vision tasks into a single framework and proposes a novel image registration and fusion method, named Super Fusion. Specifically, we design a registration network to estimate bidirectional deformation fields to rectify geometric distortions of input images under the supervision of both photometric and end-point constraints. The registration and fusion are combined in a symmetric scheme, in which while mutual promotion can be achieved by optimizing the naive fusion loss, it is further enhanced by the mono-modal consistent constraint on symmetric fusion outputs. In addition, the image fusion network is equipped with the global spatial attention mechanism to achieve adaptive feature integration. Moreover, the semantic constraint based on the pre-trained segmentation model and Lovasz-Softmax loss is deployed to guide the fusion network to focus more on the semantic requirements of high-level vision tasks. Extensive experiments on image registration, image fusion,and semantic segmentation tasks demonstrate the superiority of our Super Fusion compared to the state-of-the-art alternatives.The source code and pre-trained model are publicly available at https://github.com/Linfeng-Tang/Super Fusion.

Using shape contexts method for registration of contra lateral breasts in thermal images

AIM: To achieve symmetric boundaries for left and right breasts boundaries in thermal images by registration. METHODS: The proposed method for registration consists of two steps. In the first step, shape context, an approach as presented by Belongie and Malik was applied for registration of two breast boundaries. The shape context is an approach to measure shape similarity. Two sets of finite sample points from shape contours of two breasts are then presented. Consequently, the correspondences between the two shapes are found. By finding correspondences, the sample point which has the most similar shape context is obtained. RESULTS: In this study, a line up transformation which maps one shape onto the other has been estimated in order to complete shape. The used of a thin plate spline permitted good estimation of a plane transformation which has capability to map unselective points from one shape onto the other. The obtained aligningtransformation of boundaries points has been applied successfully to map the two breasts interior points. Some of advantages for using shape context method in this work are as follows:(1) no special land marks or key points are needed;(2) it is tolerant to all common shape deformation; and(3) although it is uncomplicated and straightforward to use, it gives remarkably powerful descriptor for point sets significantly upgrading point set registration. Results are very promising. The proposed algorithm was implemented for 32 cases. Boundary registration is done perfectly for 28 cases.CONCLUSION: We used shape contexts method that is simple and easy to implement to achieve symmetric boundaries for left and right breasts boundaries in thermal images.

Incoherent digital holographic spectral imaging with high accuracy of image pixel registration

Fresnel incoherent correlation holography(FINCH) is a unique three-dimensional(3D) imaging technique which has the advantages of scanning-free,high resolution,and easy matching with existing mature optical systems.In this article,an incoherent digital holographic spectral imaging method with high accuracy of spectral reconstruction based on liquid crystal tunable filter(LCTF) and FINCH is proposed.Using the programmable characteristics of spatial light modulator(SLM),a series of phase masks,none of whose focal lengths changes with wavelength,is designed and made.For each wavelength of LCTF output,SLM calls three phase masks with different phase constants at the corresponding wavelength,and CCD records three holograms.The spectral images obtained by this method have a constant magnification,which can achieve pixel-level image registration,restrain image registration errors,and improve spectral reconstruction accuracy.The results show that this method can not only obtain the 3D spatial information and spectral information of the object simultaneously,but also have high accuracy of spectral reconstruction and excellent color reproducibility.

A NEW IMAGE REGISTRATION METHOD FOR GREY IMAGES

The proposed algorithm relies on a group of new formulas for calculating tangent slope so as to address angle feature of edge curves of image. It can utilize tangent angle features to estimate automatically and fully the rotation parameters of geometric transform and enable rough matching of images with huge rotation difference. After angle compensation, it can search for matching point sets by correlation criterion, then calculate parameters of affine transform, enable higher-precision emendation of rotation and transferring. Finally, it fulfills precise matching for images with relax-tense iteration method. Compared with the registration approach based on wavelet direction-angle features, the matching algorithm with tangent feature of image edge is more robust and realizes precise registration of various images. Furthermore, it is also helpful in graphics matching.

THE STUDY ON AUTOMATIC REGISTRATION OF INSAR IMAGES

Interferometric Synthetic Aperture Radar (InSAR) allows production of high resolution DEM and detection of small earth motions using multiple pass SAR data sets obtained by remote sensing satellite. But the whole process has not yet reached sufficient robustness to warrant automated DEM production as commonly produced by stereovision with optical images. The automatic algorithm for precision registration is one of the bottlenecks in InSAR data processing. In this paper, an automatic approach with multi-step image matching algorithm is presented. All procedures are automatically implemented. The experiment is carried out successfully with SIR-C/L-band InSAR data. The triangular piecewise rectification is also advanced in reducing local distortion between the images and processing the large scene image. The primary result has prospect in the precision registration for the repeat-track InSAR data and reveals the potential of the presented automatic strategy.

Total Variation Constrained Non-Negative Matrix Factorization for Medical Image Registration

This paper presents a novel medical image registration algorithm named total variation constrained graphregularization for non-negative matrix factorization(TV-GNMF).The method utilizes non-negative matrix factorization by total variation constraint and graph regularization.The main contributions of our work are the following.First,total variation is incorporated into NMF to control the diffusion speed.The purpose is to denoise in smooth regions and preserve features or details of the data in edge regions by using a diffusion coefficient based on gradient information.Second,we add graph regularization into NMF to reveal intrinsic geometry and structure information of features to enhance the discrimination power.Third,the multiplicative update rules and proof of convergence of the TV-GNMF algorithm are given.Experiments conducted on datasets show that the proposed TV-GNMF method outperforms other state-of-the-art algorithms.

Multi-sensor image registration using multi-resolution shape analysis

Multi-sensor image registration has been widely used in remote sensing and medical image field, but registration performance is degenerated when heterogeneous images are involved. An image registration method based on multi-resolution shape analysis is proposed in this paper, to deal with the problem that the shape of similar objects is always invariant. The contours of shapes are first detected as visual features using an extended contour search algorithm in order to reduce effects of noise, and the multi-resolution shape descriptor is constructed through Fourier curvature representation of the contour’s chain code. Then a minimum distance function is used to judge the similarity between two contours. To avoid the effect of different resolution and intensity distribution, suitable resolution of each image is selected by maximizing the consistency of its pyramid shapes. Finally, the transformation parameters are estimated based on the matched control-point pairs which are the centers of gravity of the closed contours. Multi-sensor Landsat TM imagery and infrared imagery have been used as experimental data for comparison with the classical contour-based registration. Our results have been shown to be superior to the classical ones.

Image registration of the human accommodating eye demonstrates equivalent increases in lens equatorial radius and central thickness

AIM: To compare the results of in vivo human high resolution image registration studies of the eye during accommodation to the predictions of mathematical and finite element models of accommodation. METHODS: Data from published high quality image registration studies of pilocarpine induced accommodative changes of equatorial lens radius(ELR) and central lens thickness(CLT) were statistically analyzed. RESULTS: The mean changes in ELR and CLT were 6.76 μm/diopter and 6.51 μm/diopter, respectively. The linear regressions, reflecting the association between ELR and accommodative amplitude(AAELR) was: slope=6.58 μm/diopter, r^2=0.98, P<0.0001 and between CLT and AACLT was: slope=6.75 μm/diopter, r^2=0.83, P<0.001. On the basis of these relationships, the CLT slope and the AAELR were used to predict the measured change in ELR(ELRpredicted). There was no statistical difference between ELRpredicted and the measured ELR as demonstrated by a Student's paired t-test: P=0.96 and linear regression analysis: slope=0.97, r^2=0.98, P<0.00001.CONCLUSION: Image registration with invariant positional references demonstrates that ELR and CLT equivalently minimally increase ~7.0 μm/diopter during accommodation. The small equivalent increases in ELRand CLT are associated with a large accommodative amplitude. These findings are consistent with the predictions of mathematical and finite element models that specified the stiffness of the lens nucleus is the same or greater than the lens cortex and that accommodation involves a small force(<5 g).