增广模态收益动态评估方法

Dynamic evaluation method for benefit of modality augmentation

针对获取新模态难度大、收益差异大的问题,提出了一种增广模态收益动态评估方法。首先,通过多模态融合网络得到中间特征表示和模态融合前后的预测结果;其次,将两个预测结果的真实类别概率(TCP)引入置信度估计,得到融合前后的置信度;最后,计算两种置信度的差异,并将该差异作为样本以获取新模态所带来的收益。在常用多模态数据集和真实的医学数据集如癌症基因组图谱(TCGA)上进行实验。在TCGA数据集上的实验结果表明,与随机收益评估方法和基于最大类别概率(MCP)的方法相比,所提方法的准确率分别提高了1.73~4.93和0.43~4.76个百分点,有效样本率(ESR)分别提升了2.72~11.26和1.08~25.97个百分点。可见,所提方法能够有效评估不同样本获取新模态所带来的收益,并具备一定可解释性。

Focused on the difficulty and big benefit difference in acquiring new modalities,a method for dynamically evaluating benefit of modality augmentation was proposed.Firstly,the intermediate feature representation and the prediction results before and after modality fusion were obtained through the multimodal fusion network.Then,the confidence before and after fusion were obtained by introducing the True Class Probability(TCP)of two prediction results to confidence estimation.Finally,the difference between two confidences was calculated and used as an sample to obtain the benefit brought by the new modality.Extensive experiments were conducted on commonly used multimodal datasets and real medical datasets such as The Cancer Genome Atlas(TCGA).The experimental results on TCGA dataset show that compared with the random benefit evaluation method and the Maximum Class Probability(MCP)based method,the proposed method has the accuracy increased by 1.73 to 4.93 and 0.43 to 4.76 percentage points respectively,and the Effective Sample Rate(ESR)increased by 2.72 to 11.26 and 1.08 to 25.97 percentage points respectively.It can be seen that the proposed method can effectively evaluate benefits of acquiring new modalities for different samples,and has a certain degree of interpretability.