基于文本引导下的多模态医学图像分析算法

A Multi-Modal Medical Image Analysis Algorithm Based on Text Guidance

结合胃镜超声和白光内镜可以更准确地识别胃肠道间质瘤.但是现有的多模态方法往往仅关注于图像特征,忽略了诊断文本信息中所包含的语义信息对于精确理解和诊断医学图像的重要性.为此,本文提出一种新的基于文本引导下的多模态医学图像分析算法框架(Text-guided Multi-modal Medical image analysis framework,TMM-Net).TMM-Net使用多阶段的诊断文本来引导模型学习,以提取图像中的关键诊断信息特征,然后通过交叉模态注意力机制促进多模态特征之间的交互.值得注意的是,TMM-Net通过预测病变属性来模拟临床诊断过程,从而增强了可解释性.验证实验在两个中心包含10 025个模态数据对的数据集上进行.结果表明,该方法相比目前最优的GISTs诊断方法精度提升7.7%,同时获得了最高的(Area Under the Curve,AUC)值:0.927,其可解释性可以更好地适合临床需求.

Combining gastroscopy ultrasound and white light endoscopy can improve the accuracy of identifying gas⁃trointestinal stromal tumors(GISTs).However,existing multi-modal methods often focus solely on image features and over⁃look the semantic relevance contained in diagnostic textual information,which is crucial for precise understanding and diag⁃nosis of medical images.To address this issue,we propose a novel text-guided multi-modal medical image analysis frame⁃work(TMM-Net).TMM-Net extracts key diagnostic information features from images through a multi-stage guided model of diagnostic text,and then promotes the interaction of multi-modal features through cross-modal attention mechanisms.Nota⁃bly,TMM-Net simulates the clinical diagnostic process by predicting lesion attributes,enhancing interpretability.Validation experiments were conducted on a dataset consisting of 10025 modality data pairs from two centers.The results show that the proposed method achieves a 7.7%improvement in accuracy compared to the current state-of-the-art GISTs diagnostic meth⁃od,with the highest AUC(Area Under the Curve)value of 0.927,and its interpretability may better suit clinical needs.