注意力机制引导的多模态心脏图像分割

Attention-Guided Multimodal Cardiac Segmentation

为有效挖掘模态间共享与模态特有的信息,本文提出一种注意力机制引导的半孪生网络,用于分割多模态(MRI与CT)心脏图像.具体地,首先运用循环一致的生成对抗网络(CycleGAN)进行双向的图像生成(即从MRI到CT以及从CT到MRI),这样可以解决模态间心脏图像不配对的问题;其次,设计一个新的半孪生网络,将原始的CT(或MR)图像及其生成的MR(或CT)图像进行配对并同时输入,先通过两个编码器(encoders)分别学习模态特有的特征,再经过一个跨模态的注意力模块将不同模态的特征进行融合,最后输入一个公共的解码器(decoder)来得到模态共享的特征,用于心脏图像分割.上述学习过程是端到端的方式进行训练.本文将所提方法在真实的CT与MR不配对的心脏图像数据集上进行实验评估,表明所提方法的分割精度超出基准方法.

With the goal of leveraging the modal-shareable and modal-specific information during cross-modal segmentation,we propose a novel cross-modal attention-guided semi-Siamese network for joint cardiac segmentation from MR and CT images.In particular,we first employed the cycle-consistency generative adversarial networks to complete the bidirectional image generation(i.e.,MR to CT,CT to MR)to help reduce the modal-level inconsistency.Then,with the generated and original MR and CT images,a novel semi-Siamese network is utilized where 1)two encoders learn modal-specific features separately and 2)a common decoder makes full use of modal-shareable information from different modalities for a final consistent segmentation.Also,we implement the cross-modal attention to incorporate these shareable and specific information,and our model can be trained in an end-to-end manner.With extensive evaluation on the unpaired CT and MR cardiac images,our method outperforms the baselines in terms of the segmentation performance.