中心聚集效应下多种预测模型构建策略的模拟比较

Simulation comparison of various prediction model construction strategies under clustering effect

目的利用多中心数据构建临床预测模型时,数据的独立性假设会发生违背,研究对象之间存在明显中心聚集效应,为了充分考虑聚集性问题,本研究拟比较考虑中心聚集效应的随机截距Logistic回归模型(RI)和固定效应模型(FEM)与不考虑中心聚集效应的标准Logistic回归模型(SLR)和随机森林算法(RF)在不同场景下的模型性能。方法模拟预测模型建立过程中,存在不同程度中心聚集效应时,在中心水平上不同模型的预测性能,包括在不同场景中的区分度和校准度差异,同时比较这种差异在不同事件率时的变化趋势。结果在中心水平,不同模型(除RF外)在中心聚集效应下不同场景的区分度差异不大,其C-index均值变化很小。利用多中心高度聚集的数据进行预测时,边缘预测(M.RI、SLR和RF)与条件预测相比校准截距略小于0,高估了预测的平均概率。其中RF则在多中心大样本条件下截距校准表现很好,这也体现了机器学习算法对处理大样本数据的优势。在中心多患者少时,FEM进行条件预测,校准截距大于0,预测的平均概率被低估。此外,在利用多中心大样本数据开发预测模型时,三个条件预测(FEM、A.RI、C.RI)斜率校准较好,边缘预测(M.RI和SLR)的校准斜率大于1出现了欠拟合的问题,且随着中心聚集效应增加欠拟合问题越发凸显。特别是在中心少患者少时,数据的过拟合会掩盖边缘预测与条件预测校准性能上的差异。最后,越低的事件发生率时,中心聚集效应在中心水平对不同模型预测性能的影响越明显。结论利用高度聚集的多中心数据构建模型并应用于特定环境中预测,当中心数较少或因不同发病率导致中心间差异较大时可以选择RI和FEM进行条件预测;当中心数较多、样本量较大时可选择RI进行条件预测或RF进行边缘预测。

Objective When using multi-center data to construct clinical prediction models,the independence assumption of data will be violated,and there is an obvious clustering effect among research objects.In order to fully consider the clustering effect,this study intends to compare the model performance of the random intercept logistic regression model(RI)and the fixed effects model(FEM)considering the clustering effect with the standard logistic regression model(SLR)and the random forest algorithm(RF)without considering the clustering effect under different scenarios.Methods In the process of forecasting model establishment,the prediction performance of different models at the center level was simulated when there were different degrees of clustering effects,including the difference of discrimination and calibration in different scenarios,and the change trend of this difference at different event rates was compared.Results At the center level,different models,except RF,showed little difference in the discrimination of different scenarios under the clustering effect,and the mean of their C-index changed very little.When using multi-center highly clustered data for forecasting,the marginal forecasts(M.RI,SLR and RF)had calibrated intercepts slightly less than 0 compared with the conditional forecasts,which overestimated the average probability of prediction.RF performed well in intercept calibration under the condition of multi-center and large samples,which also reflected the advantage of machine learning algorithm for processing large sample data.When there were few multiple patients in the center,the FEM made conditional predictions,the calibrated intercept was greater than 0,and the predicted mean probability was underestimated.In addition,when the multi-center large sample data were used to develop the prediction model,the slopes of the three conditional forecasts(FEM,A.RI,C.RI)were well calibrated,while the calibrated slopes of the marginal forecasts(M.RI and SLR)were greater than 1,which led to the problem of underfitting,and the underfitting problem became more prominent with the increase in the central aggregation effect.In particular,when there were few centers and few patients,overfitting of the data could mask the difference in calibration performance between marginal and conditional forecasts.Finally,the lower the event rate the central clustering effect at the central level had a more pronounced impact on the forecasting performance of the different models.Conclusion The highly clustered multicenter data are used to construct the model and apply it to the prediction in a specific environment.RI and FEM can be selected for conditional prediction when the number of centers is small or the difference between centers is large due to different incidence rates.When the number of hearts is large and the sample size is large,RI can be selected for conditional prediction or RF for edge prediction.