自适应多模态特征融合胶质瘤分级网络

Adaptive multi-modality fusion network for glioma grading

目的胶质瘤的准确分级是辅助制定个性化治疗方案的主要手段,但现有研究大多数集中在基于肿瘤区域的分级预测上,需要事先勾画感兴趣区域,无法满足临床智能辅助诊断的实时性需求。因此,本文提出一种自适应多模态特征融合网络(adaptive multi-modal fusion net,AMMFNet),在不需要勾画肿瘤区域的情况下,实现原始采集图像到胶质瘤级别的端到端准确预测。方法 AMMFNet方法采用4个同构异义网络分支提取不同模态的多尺度图像特征;利用自适应多模态特征融合模块和降维模块进行特征融合;结合交叉熵分类损失和特征嵌入损失提高胶质瘤的分类精度。为了验证模型性能,本文采用MICCAI(Medical Image Computing and Computer Assisted Intervention Society) 2018公开数据集进行训练和测试,与前沿深度学习模型和最新的胶质瘤分类模型进行对比,并采用精度以及受试者曲线下面积(area under curve,AUC)等指标进行定量分析。结果在无需勾画肿瘤区域的情况下,本文模型预测胶质瘤分级的AUC为0.965;在使用肿瘤区域时,其AUC高达0.997,精度为0.982,比目前最好的胶质瘤分类模型——多任务卷积神经网络同比提高1.2%。结论本文提出的自适应多模态特征融合网络,通过结合多模态、多语义级别特征,可以在未勾画肿瘤区域的前提下,准确地实现胶质瘤分级预测。

Objective Glioma grading has been a vital research tool for customized treatment in of the glioma.Glioma grading can be an assessment tool for biopsy and histopathological to resolve invasive and time-consuming issues.A non-invasive scheme for grading gliomas precision has played the key role.A reliable non-invasive grading scheme has been implemented for magnetic resonance imaging(MRI) to facilitate computer-assisted diagnosis system(CAD) for glioma grading.The medical image-based grading has been used manual features to implement image-level tumor analysis.Manual featurebased methods have realized higher area under curve(AUC) based on the variation of image intensity and image deformation analyses constrained generalization capability.The emerging deep learning method has projected deep features more semantic and representative compared with the manual features in generalization.The original data has been projected into semantic space to task-level features for data segmentation compared with image-level features.Deep feature-based models have been more qualifier than manual feature-based models on the aspect of classification tasks.Time-consuming and laborintensive segmentation has been analyzed in lesion regions for tumor.The prediction have been constrained by the tumor segmentation accuracy.A multi-modality fusion network(AMMFNet) has been applied in grading gliomas instead of tumor segmentation.Method AMMFNet has been an end-to-end multi-scale model to improve glioma grading performance via deep learning-based multi-modal fusion features.The network has contained three components including multi-modal image feature extraction module, adaptive multi-modal and multi-scale features fusion module and classification module respectively.The feature extraction module has been used for deep features extraction based on the images acquired with four different modalities.The width and depth of model has good quality in extracting semantic features.The adaptive multi-modal and multi-scale features fusing module have intended to learn fusion rules via multi-modal and multi-scale deep features.The fusion of the multi-modal features in the same semantic level via high dimensional convolution layers.An adaptive dimension reduction module have adopted to fuse the features in different semantic levels to realize the different shape of feature maps into the same size.Such reduction module has been built up in three-branch structure based on the unique dimension reducing implementation for each branch.Task-level loss and feature-level loss have been used to train the proposed model to upgrade glioma grading accuracy.The task-level loss has been achieved via weighted cross entropy.The feature-level loss has been used to maximize the intra-class feature similarity and inter-class feature discrepancy via the cosine function of two feature vectors.Based on the public Medical Image Computing and Computer Assisted Intervention Society(MICCAI)2018 challenge dataset to train and test the proposed model, the accuracy( ACC), specificity( SPE), sensitivity(SEN), positive precision value(PPV), negative precision value(NPV), average F1 score and AUC have been prior to analyze the grading validation.Result The AUC has been the highest one with a value of 0.965 comparing with visual geometry group 19-layer net( VGG19), Res Net, SENet, SERes Net, Inception V4 and NASNet.ACC, SPE, PPV,F1-score have been increased by 2.8%, 2.1%, 1.1%, and 3.1%each at most.The tumor region of interest(ROI)modeling input has been trained.The ACC has been increased by 1.2%.Ablation experiments including replacing the deeper convolutional layer with Resblock as well as adding SE block into fusion module have been further validated via customized learning modules.The ACC, SEN, SPE, PPV, NPV, F1 of AMMFNet using SE fusion block have been increased by0.9%, 0.1%, 2.5%, 1.0%, 0.6%, 1.2%respectively in the context of baseline.Conclusion The adaptive multi-modal fusion network has been demonstrated via fused multi-modal features and the multi-scale integration of fused deep features.The multi-modal and multi-scale features integration has been illustrated to capture more expressive features related to image details.The demonstration can be prior to locate the tumor even without lesion areas or segmented tumor.An endto-end model has the good priority in glioma grading.