大规模数据下基于充分降维的Leverage重要性抽样方法

Leverage Importance Sampling Based on Sufficient Dimension Reduction for Large-scale Data

随着信息技术的飞速发展,大规模数据在短时间内搜集并储存下来,为分析决策提供了巨大的信息量,也给统计建模带来了一定难度。对于样本容量大、变量个数少的数据,Leverage重要性抽样是一个简便可行的方法。本文发现,该方法中度量样本重要性的Leverage分数与因变量无关,而且在维度较大的情形下对样本没有区分程度,使得估计结果较差。为了同时考虑因变量和维度的影响,本文提出了基于充分降维的Leverage重要性抽样方法。该方法以不损失信息为前提,在充分降维的空间内重新计算Leverage分数,使得抽样更具有代表性。模拟数据分析显示,在样本容量较大的复杂数据中,相比于原始的Leverage重要性抽样方法,本文提出的方法可以降低估计的均方误差。三个实际数据也证实了该方法的可行性和有效性。

With the rapid development of information technology,large-scale data is collected and stored in a short time,which provides a huge amount of information for decision making,and also brings some difficulty to statistical modeling. For data with large n and small p,Leverage importance sampling is a convenient method. However,we find that the Leverage score is independent of the response and can’t measure the importance of samples in data with large p,which leads to poor estimation accuracy. In consideration of response and high dimension at the same time,this paper proposes a Leverage importance sampling method based on sufficient dimension reduction. With no loss of information,Leverage scores computed in reduced dimensional space are more representative. Simulated data analysis shows that,compared with the original Leverage importance sampling,the proposed method reduces the estimation error in the large-scale data. The analysis of three large datasets proves the better performance of the proposed method.