基于多模态多示例学习的免疫介导性肾小球疾病自动分类方法

Automatic classification of immune-mediated glomerular diseases based on multi-modal multi-instance learning

目的探讨如何利用多模态深度学习方法,联合光学显微镜(OM)、免疫荧光显微镜(IM)及透射电子显微镜(TEM)对应的3种图像进行免疫介导性肾小球疾病分类。方法基于273例患者的病理图像进行回顾性研究,构建多模态多示例模型对3种免疫介导性的肾小球疾病——免疫球蛋白A肾病(IgAN)、膜性肾病(MN)、狼疮性肾炎(LN)进行分类。该模型采用示例水平的多示例学习(I-MIL)方法挑选患者的TEM图像并与同一患者的OM图像和IM图像进行多模态特征融合。通过该模型与单模态、双模态模型的比较,探究3种模态之间的不同组合形式以及模态特征融合方式的特性。结果联合OM、IM以及TEM图像建立的多模态多示例模型准确率为(88.34±2.12)%,优于准确率为(87.08±4.25)%的最优的单模态模型,以及准确率为(87.92±3.06)%的最优的双模态模型。结论本研究成功建立基于OM、IM及TEM三种模态图像的多模态多示例模型,并验证了采用多示例学习结合多模态学习方法对免疫介导性肾小球疾病分类的有效性。

Objective To develop a multi-modal deep learning method for automatic classification of immune-mediated glomerular diseases based on images of optical microscopy(OM),immunofluorescence microscopy(IM),and transmission electron microscopy(TEM).Methods We retrospectively collected the pathological images from 273 patients and constructed a multi-modal multi-instance model for classification of 3 immune-mediated glomerular diseases,namely immunoglobulin A nephropathy(IgAN),membranous nephropathy(MN),and lupus nephritis(LN).This model adopts an instance-level multiinstance learning(I-MIL)method to select the TEM images for multi-modal feature fusion with the OM images and IM images of the same patient.By comparing this model with unimodal and bimodal models,we explored different combinations of the 3 modalities and the optimal methods for modal feature fusion.Results The multi-modal multi-instance model combining OM,IM,and TEM images had a disease classification accuracy of(88.34±2.12)%,superior to that of the optimal unimodal model[(87.08±4.25)%]and that of the optimal bimodal model[(87.92±3.06)%].Conclusion This multi-modal multi-instance model based on OM,IM,and TEM images can achieve automatic classification of immune-mediated glomerular diseases with a good classification accuracy.