水声目标分类算法性能评估

Performance evaluation on the algorithm of underwater acoustic target classification

水声目标分类算法研究中使用的性能指标单一且受样本不平衡影响,其评估方法也不适于样本有限的场景。针对这些问题,本文基于水声场景需求,建立了仿真模型,并分析了性能指标与样本类分布的关系,比较了不同样本条件下各评估方法的估计误差。结果表明:PR曲线下面积具有稳定的鉴别力,均衡正确率等指标受样本类分布影响小,各评估方法在样本有限时估值差异显著。据此,本文构建性能指标体系可用于设计和评估算法,提出适应水声需求的修正均衡正确率,同时推荐采用5×2分层交叉验证的评估方法。

The single performance metric used in the algorithm for underwater acoustic target classification is affected by sample imbalance and its performance evaluation method is not suitable for scenes with a limited number of samples.To address these problems,in this study,we develop simulation models based on the requirements of the underwater acoustic scenes to analyze the relationship between the performance metrics and the class distributions of the samples.We then compare the estimation errors of the evaluation methods under different sample conditions.The results show that the area under the precision-recall curve has stable discriminability,the distribution of samples has little influence on the balanced accuracy,and there is a significant difference among the evaluation methods when the sample size is limited.Accordingly,a performance metric system is established for use in designing and evaluating algorithms,a corrected balanced accuracy is proposed that is adaptable to underwater acoustic needs,and the 5×2 stratified cross-validation evaluation method is recommended.