Obtaining accurate estimates of machine learning model uncertainties on newly predicted data is essential for understanding the accuracy of the model and whether its predictions can be trusted.A common approach to suc...Obtaining accurate estimates of machine learning model uncertainties on newly predicted data is essential for understanding the accuracy of the model and whether its predictions can be trusted.A common approach to such uncertainty quantification is to estimate the variance from an ensemble of models,which are often generated by the generally applicable bootstrap method.In this work,we demonstrate that the direct bootstrap ensemble standard deviation is not an accurate estimate of uncertainty but that it can be simply calibrated to dramatically improve its accuracy.We demonstrate the effectiveness of this calibration method for both synthetic data and numerous physical datasets from the field of Materials Science and Engineering.The approach is motivated by applications in physical and biological science but is quite general and should be applicable for uncertainty quantification in a wide range of machine learning regression models.展开更多
基金The National Science Foundation provided financial support for G.P.(Award#1545481)S.D.,A.P.,J.P.E.,and X.Y.(Award#1636950 and 1636910)+1 种基金R.J.and D.M.(Award#1931298)Financial support for A.G.and G.G.was provided by the University of Wisconsin Harvey D.Spangler Professorship.
文摘Obtaining accurate estimates of machine learning model uncertainties on newly predicted data is essential for understanding the accuracy of the model and whether its predictions can be trusted.A common approach to such uncertainty quantification is to estimate the variance from an ensemble of models,which are often generated by the generally applicable bootstrap method.In this work,we demonstrate that the direct bootstrap ensemble standard deviation is not an accurate estimate of uncertainty but that it can be simply calibrated to dramatically improve its accuracy.We demonstrate the effectiveness of this calibration method for both synthetic data and numerous physical datasets from the field of Materials Science and Engineering.The approach is motivated by applications in physical and biological science but is quite general and should be applicable for uncertainty quantification in a wide range of machine learning regression models.
