Cross Validation Based Model Averaging for Varying-Coefficient Models with Response Missing at Random

In this paper, a model averaging method is proposed for varying-coefficient models with response missing at random by establishing a weight selection criterion based on cross-validation. Under certain regularity conditions, it is proved that the proposed method is asymptotically optimal in the sense of achieving the minimum squared error.

Testing conditional independence with data missing at random

It is known that conditional independence is a quite basic assumption in many fields of statistics. How to test its validity is of great importance and has been extensively studied by the literature. Nevertheless, all of the existing methods focus on the case that data are fully observed, but none of them seems having taken into account of the scenario when missing data are present. Motivated by this, this paper develops two testing statistics to handle such a situation relying on the idea of inverse probability weighted and augmented inverse probability weighted techniques. The asymptotic distributions of the proposed statistics are also derived under the null hypothesis. The simulation studies indicate that both testing statistics perform well in terms of size and power.

Generalized unscented Kalman filtering based radial basis function neural network for the prediction of ground radioactivity time series with missing data

On the assumption that random interruptions in the observation process are modeled by a sequence of independent Bernoulli random variables, we firstly generalize two kinds of nonlinear filtering methods with random interruption failures in the observation based on the extended Kalman filtering （EKF） and the unscented Kalman filtering （UKF）, which were shortened as GEKF and CUKF in this paper, respectively. Then the nonlinear filtering model is established by using the radial basis function neural network （RBFNN） prototypes and the network weights as state equation and the output of RBFNN to present the observation equation. Finally, we take the filtering problem under missing observed data as a special case of nonlinear filtering with random intermittent failures by setting each missing data to be zero without needing to pre-estimate the missing data, and use the GEKF-based RBFNN and the GUKF-based RBFNN to predict the ground radioactivity time series with missing data. Experimental results demonstrate that the prediction results of GUKF-based RBFNN accord well with the real ground radioactivity time series while the prediction results of GEKF-based RBFNN are divergent.

CBPS-Based Inference in Nonlinear Regression Models with Missing Data

In this article, to improve the doubly robust estimator, the nonlinear regression models with missing responses are studied. Based on the covariate balancing propensity score (CBPS), estimators for the regression coefficients and the population mean are obtained. It is proved that the proposed estimators are asymptotically normal. In simulation studies, the proposed estimators show improved performance relative to usual augmented inverse probability weighted estimators.

Using Statistical Learning to Treat Missing Data: A Case of HIV/TB Co-Infection in Kenya

In this study, we investigate the effects of missing data when estimating HIV/TB co-infection. We revisit the concept of missing data and examine three available approaches for dealing with missingness. The main objective is to identify the best method for correcting missing data in TB/HIV Co-infection setting. We employ both empirical data analysis and extensive simulation study to examine the effects of missing data, the accuracy, sensitivity, specificity and train and test error for different approaches. The novelty of this work hinges on the use of modern statistical learning algorithm when treating missingness. In the empirical analysis, both HIV data and TB-HIV co-infection data imputations were performed, and the missing values were imputed using different approaches. In the simulation study, sets of 0% (Complete case), 10%, 30%, 50% and 80% of the data were drawn randomly and replaced with missing values. Results show complete cases only had a co-infection rate (95% Confidence Interval band) of 29% (25%, 33%), weighted method 27% (23%, 31%), likelihood-based approach 26% (24%, 28%) and multiple imputation approach 21% (20%, 22%). In conclusion, MI remains the best approach for dealing with missing data and failure to apply it, results to overestimation of HIV/TB co-infection rate by 8%.

基于MAR-MRF的SAR图像分割方法

该文提出了一种基于多尺度自回归模型和马尔科夫随机场的SAR图像分割算法。算法引入多尺度自回归模型,建立层与层之间以及相邻层的像素点之间的数学关系,并将此模型与马尔科夫分割算法结合,实现了更为合理的多尺度分割策略。通过相邻尺度的依赖关系及同一尺度空间的马尔可夫性,使用多尺度自回归模型的预测结果来引导精细尺度图像分割,不仅使得最细尺度下的分割迭代次数减少;而且去除了最细尺度下多余的误分类斑块;同时还能够分割出清晰、平滑的目标边界,实现了较满意的SAR图像分割。

Random Subspace Sampling for Classification with Missing Data

Many real-world datasets suffer from the unavoidable issue of missing values,and therefore classification with missing data has to be carefully handled since inadequate treatment of missing values will cause large errors.In this paper,we propose a random subspace sampling method,RSS,by sampling missing items from the corresponding feature histogram distributions in random subspaces,which is effective and efficient at different levels of missing data.Unlike most established approaches,RSS does not train on fixed imputed datasets.Instead,we design a dynamic training strategy where the filled values change dynamically by resampling during training.Moreover,thanks to the sampling strategy,we design an ensemble testing strategy where we combine the results of multiple runs of a single model,which is more efficient and resource-saving than previous ensemble methods.Finally,we combine these two strategies with the random subspace method,which makes our estimations more robust and accurate.The effectiveness of the proposed RSS method is well validated by experimental studies.

Smoothed Empirical Likelihood Inference for Nonlinear Quantile Regression Models with Missing Response

In this paper, three smoothed empirical log-likelihood ratio functions for the parameters of nonlinear models with missing response are suggested. Under some regular conditions, the corresponding Wilks phenomena are obtained and the confidence regions for the parameter can be constructed easily.

矿井通风参数缺失数据插补方法

矿井智能通风系统对矿山智能化建设至关重要。为解决矿井通风参数在实际测量时,因为巷道不具备测试条件、仪器信号受到干扰、巷道断面风速不均一、人工操作不当等制约性因素,造成的矿井通风参数数据缺失问题,提出了1种基于随机森林−链式方程多重插补法的矿井通风参数缺失数据插补方法。采用链式方程多重插补法,通过迭代对每个缺失的属性值产生n个插补值,从而产生n个完整数据集,对n个完整数据集进行分析优化得到1个最终的完整数据集。为了提高缺失值插补精度,合理考虑了矿井通风参数缺失数据的不确定性对分析过程的影响,在随机森林的预测任务中,结合预测均值匹配模型对缺失数据进行插补。以潞新二矿为实验对象,利用智能矿井通风仿真系统IMVS对潞新二矿矿井通风参数原始数据集进行数据预处理,得到完整、准确的矿井通风参数完整数据集,对完整数据集分别进行了不同缺失属性、不同数据缺失率、不同迭代次数的对比试验。以多种模型评价指标对模型有效性进行评估。结果表明:基于随机森林的链式方程多重插补模型插补形成的完整数据集与原始数据集具有很好的相似性;对不同缺失列进行插补实验的结果显示插补模型可以轻松处理混合类型的数据,自主学习参数之间的相关性从而降低了插补复杂性;迭代后形成的n个数据集通过分析合并成一个最终数据集,提高了插补准确率;对初始插补后的完整数据集进行不同迭代次数的试验,发现迭代超过一定次数后,数据相关性一定会收敛。

缺失数据过程的自适应多元EWMA控制图

随着生产过程的日益复杂,多元统计过程控制(SPC)领域对在线算法的关注与日俱增.然而,基于完整数据和均匀时间间隔假设的传统方法在存在缺失数据时表现并不理想.为了最大化利用可用信息,我们提出了一种自适应指数加权移动平均(EWMA)控制图,它采用了加权插补方法,能够充分利用完整数据和不完整数据之间的关系.具体而言,我们首先引入了两种恢复方法:改进的K近邻方法和传统的单变量EWMA方法.然后,我们构造了一个自适应加权函数来结合这两种方法,即当样本信息表明过程超出控制的可能性增加时,会降低EWMA统计量的权重,反之亦然.通过模拟结果和一个实际案例,我们证明了所提出方案的稳健性和敏感性.

不可忽略的无响应缺失下的协变量选择

本文旨在建立一个存在不可忽略的无响应缺失时高维协变量向量的协变量选择方法.由于有不可忽略的缺失响应数据,必须建立一种新的协变量选择方法来删除既与响应变量也与缺失机制无关的协变量.一旦冗余协变量被删除,现有的缺失机制估计和其他基于逆缺失机制加权的分析方法可以被应用.我们提供了一些模拟结果来展示我们方法的有效性.

基于随机森林和最近邻插值法的交通流量数据修复方法

针对目前传感器在采集数据过程中由于受到天气或者自身设备故障等原因,造成数据缺失或者数据异常,导致不能从采集的数据中获得准确的交通变化规律等问题,分别提出基于改进最近邻插值算法和基于随机森林插补的交通流量数据缺失修复模型。由于交通数据缺失场景和缺失类型以及时空关联的差异性,将数据缺失类型划分为简单随机缺失和复杂连续缺失两种;利用改进的最近邻插值算法建立模型处理简单随机缺失,建立随机森林模型进行迭代插补处理复杂连续缺失;面对两种不同的数据缺失类型,利用期望最大化算法、深度信念网络、季节性差分自回归滑动平均模型分别搭建模型对比交叉验证改进的最近邻插值算法和随机森林插补方法。数据来源于美国加利福尼亚州PeMS(performance measurement system)实时采集的2022年6月1日—2022年7月31日以5 min为采样时间间隔的交通流量数据,为了模拟数据的缺失状况,将完整数据按照一定比例进行缺失,来模拟数据缺数的情况,得到简单随机缺失和复杂连续缺失分布的交通流量缺失数据集。结果表明:本实验在不同的缺失比例下均有良好的表现,通过设计不同的缺失比例和类型,各项评估指标均有明显优势,验证了两种数据缺失填充模型的有效性。

基于随机森林回归算法的抽油机井系统效率分析与预测

抽油机井系统效率过低,则无功损耗大,必然造成能耗的浪费,因此有必要对系统效率进行分析研究。首先根据系统效率计算公式进行分析,构建了12类属性指标数据集,用于有功功率的分析和回归;采用随机森林回归算法,对指标数据集进行训练集回归,并对测试集测试;最后,采用随机森林回归算法对现场的抽油机井系统效率进行了预测。对训练集2560口抽油机井进行回归,得出系统效率主要受日产液量影响,其次为有功功率,二者重要性占72.2%;全样本特征属性预测和缺样本特征属性预测的测试集的确定系数分别为0.852和0.701,说明在有功功率缺失时,拟合质量降低,但系统效率的变异中可由各属性指标参数解释部分的占比仍较大;根据缺样本特征属性预测回归模型,在现场对系统效率低于15%的188口井进行措施调整,累计节电15.28×10^(4)kWh,折合经济效益9.73万元。

应用小波通道注意力网络的地震数据重建方法

重建缺失的地震道是地震数据处理的关键环节之一。近年来提出了多种基于深度学习理论的地震数据重建方法。然而,这些方法中常用的卷积运算只能捕捉到地震数据的局部特征,没有充分利用全局信息。另外,池化操作也会造成特征图信息的丢失,从而破坏地震反射的细节特征。为此,提出了基于小波通道注意力网络的地震数据重建方法。哈尔(Haar)小波变换能够有效提取信号的多尺度特征,并在上采样过程中避免信息的丢失;高效通道注意力模块通过对不同通道特征图之间的相关性进行建模,能实现全局信息的充分利用。合成和实际地震数据的实验结果表明,与具有代表性的深度学习方法相比,文中所提出的网络模型可以产生更准确的重建结果。

存在非随机缺失数据的纵向数据中介分析

纵向中介分析面临两个挑战:一是某个时刻的中介和结果会影响后续时刻的中介和结果,从而成为治疗后混淆因子(也被称作时变混淆因子);二是非随机缺失数据在纵向研究中很常见,如果没有处理好则会带来系统的中介效应估计偏差。目前没有文献考虑纵向中介分析同时出现治疗后混淆和非随机缺失数据的情形。为了填补这一空缺,基于潜在结果模型框架的理论,本文提出了纵向中介分析存在非随机缺失数据时因果效应的可识别性条件。开发了一种新的估计中介效应的统计程序,使用估计方程方法和多重插补处理缺失数据,并使用修正后的自然效应模型来估计中介效应。建立了所提出方法的大样本性质,并通过随机模拟和实际数据分析评估了新方法的有限样本表现。

基于缺失数据填补的油浸式变压器故障诊断

数据质量是影响变压器故障诊断模型准确性及可靠性的重要因素。针对现有变压器故障诊断模型对数据完整性要求较高等问题,以油浸式变压器为研究对象,提出了一种基于缺失数据填补的变压器故障诊断方法。首先,采用极端随机树(extremely randomized trees,ERT)算法对变压器样本的缺失数据进行填补,通过与多种回归模型对比,评价ERT模型的预测效果。然后,基于油中溶解气体数据,提取能够反映变压器运行状态的16维特征集合,形成完备信息的变压器故障诊断样本。最后,利用树结构概率密度估计(tree-structured parzen estimator,TPE)算法实现梯度提升树(gradient boosting decision tree,GBDT)模型的参数优化,构建基于TPE-GBDT的变压器故障诊断模型。研究结果表明,在对缺失率为10%的变压器样本数据进行填补时,ERT算法的决定系数达到0.96,高于线性回归和随机森林回归等算法。此外,基于ERT填补后的样本数据在TPE-GBDT模型的平均诊断准确率和标准差分别为90.1%和0.036,其准确性和稳定性均优于线性判别分析和随机森林分类等算法。该方法能够有效提升变压器样本质量和故障诊断效果,可为变压器运维检修提供针对性的指导建议。

一种顾及空间异质性和噪声的遥感缺失数据重建方法

针对光学遥感数据常存在大量缺失数据和噪声,以及现有光学遥感缺失数据重建算法大都未充分顾及地理数据空间相关密切程度的问题,本文充分利用地理空间数据间的时空关联性,提出了一种协同随机森林(RF)和地理加权回归(GWR)的重建方法(RF+GWR),分别以GF-4归一化植被指数(NDVI)、MODIS地表温度(LST)和GF-4反射率数据为试验材料,对RF+GWR方法的普适性和缺失重建性能进行了评估。试验结果表明,在所设不同云量掩膜水平下,相比于KNN和RF,RF+GWR方法在GF-4 NDVI、MODIS LST和GF-4波段反射率缺失数据方面的重建性能均有不同程度的改善,均方根误差、平均绝对误差和决定系数最大提升分别为33.07%、30.19%和7.06%。

基于联合树的高维数据本地化差分隐私保护算法

为解决发布高维数据过程中复杂的属性关联问题并避免中心服务器不可信任的问题,提出一种基于联合树的高维数据本地化差分隐私保护算法(JT-LDP算法)。基于不可信的中心服务器实现对用户数据的本地化差分隐私保护,中心服务器接收到用户本地化差分隐私保护的数据后,基于联合树算法识别高维数据的属性相关性,将高维数据属性集分割成多个独立的低维属性集。通过采样合成新的数据集进行发布。实验结果表明,JT-LDP算法在高维数据情况下具有更高的精度。

基于函数型数据模型的PM2.5浓度与温度关系的实证分析

考虑在函数型解释变量部分观测的情况下,用函数线性模型刻画与标量响应变量的关系.基于函数型主成分分析(Functional Principal Component Analysis,简称FPCA)实现了对缺失部分样本的重构,并通过实证分析,对一组北京市2010-2014年间统计的包括部分观测PM2.5数值的气象数据,分析了PM2.5作为部分观测函数型解释变量对标量响应变量平均气温的影响,结果表明了该方法具有处理缺失函数数据的现实意义.

A Missing Power Data Filling Method Based on Improved Random Forest Algorithm

Missing data filling is a key step in power big data preprocessing,which helps to improve the quality and the utilization of electric power data.Due to the limitations of the traditional methods of filling missing data,an improved random forest filling algorithm is proposed.As a result of the horizontal and vertical directions of the electric power data are based on the characteristics of time series.Therefore,the method of improved random forest filling missing data combines the methods of linear interpolation,matrix combination and matrix transposition to solve the problem of filling large amount of electric power missing data.The filling results show that the improved random forest filling algorithm is applicable to filling electric power data in various missing forms.What’s more,the accuracy of the filling results is high and the stability of the model is strong,which is beneficial in improving the quality of electric power data.