Shrinkage Estimation of Semiparametric Model with Missing Responses for Cluster Data

This paper simultaneously investigates variable selection and imputation estimation of semiparametric partially linear varying-coefficient model in that case where there exist missing responses for cluster data. As is well known, commonly used approach to deal with missing data is complete-case data. Combined the idea of complete-case data with a discussion of shrinkage estimation is made on different cluster. In order to avoid the biased results as well as improve the estimation efficiency, this article introduces Group Least Absolute Shrinkage and Selection Operator (Group Lasso) to semiparametric model. That is to say, the method combines the approach of local polynomial smoothing and the Least Absolute Shrinkage and Selection Operator. In that case, it can conduct nonparametric estimation and variable selection in a computationally efficient manner. According to the same criterion, the parametric estimators are also obtained. Additionally, for each cluster, the nonparametric and parametric estimators are derived, and then compute the weighted average per cluster as finally estimators. Moreover, the large sample properties of estimators are also derived respectively.

Estimation and test of restricted linear EV model with nonignorable missing covariates

This paper deals with estimation and test procedures for restricted linear errors-invariables(EV) models with nonignorable missing covariates. We develop a restricted weighted corrected least squares(WCLS) estimator based on the propensity score, which is fitted by an exponentially tilted likelihood method. The limiting distributions of the proposed estimators are discussed when tilted parameter is known or unknown. To test the validity of the constraints,we construct two test procedures based on corrected residual sum of squares and empirical likelihood method and derive their asymptotic properties. Numerical studies are conducted to examine the finite sample performance of our proposed methods.

A Structural Equation Model (SEM) for Pharmacist Competencies in Improving Quality of Life of Cancer Patients: Effect of Missing Values on the SEM

Objective: With the goal of improving health-related quality of life (HRQOL) in cancer patients, we previously reported a structural equation model (SEM) of subjected QOL and qualifications of pharmacists, based on a series of questionnaires completed by patients and pharmacists. However, several patients and pharmacists were excluded from the previous study because it was not always possible to obtain all the data intended for collection. In order to reveal the effect of missing data on the SEM, we established SEMs of HRQOL and the competency of pharmacists, using correlation matrices derived by two different statistical methods for handling missing data. Method: Fifteen cancer patients hospitalized for cancer and were receiving opioid analgesics for pain control, and eight pharmacists were enrolled in this study. Each subject was asked four times weekly to answer questions presented in a questionnaire. SEMs were explored using two correlation matrices derived with pair-wise deletion (PD matrix) and list-wise deletion (LD matrix). The final models were statistically evaluated with certain goodness-of-fit criteria. Results: Data were intended to be collected four times weekly for each patient, but there were some missing values. The same SEMs for HRQOL were optimized using both the LD and PD matrices. Although the path diagrams of the SEMs were not identical in the “competency of pharmacists,” the two models suggested that a higher competency of a pharmacist lowered the “severity” of condition and increased the “comfort” of patients, resulting in an increase in the subjected QOL. Conclusion: In collecting data for clinical research, missing values are unavoidable. When the structure of the model was robust enough, the missing data had a minor effect on our SEM of QOL. In QOL research, the LD matrix as well as the PD matrix would be effective, provided the model is sufficiently robust.

Inverse Bayesian Estimation of Gravitational Mass Density in Galaxies from Missing Kinematic Data

In this paper, we focus on a type of inverse problem in which the data are expressed as an unknown function of the sought and unknown model function (or its discretised representation as a model parameter vector). In particular, we deal with situations in which training data are not available. Then we cannot model the unknown functional relationship between data and the unknown model function (or parameter vector) with a Gaussian Process of appropriate dimensionality. A Bayesian method based on state space modelling is advanced instead. Within this framework, the likelihood is expressed in terms of the probability density function (pdf) of the state space variable and the sought model parameter vector is embedded within the domain of this pdf. As the measurable vector lives only inside an identified sub-volume of the system state space, the pdf of the state space variable is projected onto the space of the measurables, and it is in terms of the projected state space density that the likelihood is written;the final form of the likelihood is achieved after convolution with the distribution of measurement errors. Application motivated vague priors are invoked and the posterior probability density of the model parameter vectors, given the data are computed. Inference is performed by taking posterior samples with adaptive MCMC. The method is illustrated on synthetic as well as real galactic data.

Study on the Missing Data Mechanisms and Imputation Methods

The absence of some data values in any observed dataset has been a real hindrance to achieving valid results in statistical research. This paper aimed at the missing data widespread problem faced by analysts and statisticians in academia and professional environments. Some data-driven methods were studied to obtain accurate data. Projects that highly rely on data face this missing data problem. And since machine learning models are only as good as the data used to train them, the missing data problem has a real impact on the solutions developed for real-world problems. Therefore, in this dissertation, there is an attempt to solve this problem using different mechanisms. This is done by testing the effectiveness of both traditional and modern data imputation techniques by determining the loss of statistical power when these different approaches are used to tackle the missing data problem. At the end of this research dissertation, it should be easy to establish which methods are the best when handling the research problem. It is recommended that using Multivariate Imputation by Chained Equations (MICE) for MAR missingness is the best approach to dealing with missing data.

Improving Disease Prevalence Estimates Using Missing Data Techniques

The prevalence of a disease in a population is defined as the proportion of people who are infected. Selection bias in disease prevalence estimates occurs if non-participation in testing is correlated with disease status. Missing data are commonly encountered in most medical research. Unfortunately, they are often neglected or not properly handled during analytic procedures, and this may substantially bias the results of the study, reduce the study power, and lead to invalid conclusions. The goal of this study is to illustrate how to estimate prevalence in the presence of missing data. We consider a case where the variable of interest (response variable) is binary and some of the observations are missing and assume that all the covariates are fully observed. In most cases, the statistic of interest, when faced with binary data is the prevalence. We develop a two stage approach to improve the prevalence estimates;in the first stage, we use the logistic regression model to predict the missing binary observations and then in the second stage we recalculate the prevalence using the observed data and the imputed missing data. Such a model would be of great interest in research studies involving HIV/AIDS in which people usually refuse to donate blood for testing yet they are willing to provide other covariates. The prevalence estimation method is illustrated using simulated data and applied to HIV/AIDS data from the Kenya AIDS Indicator Survey, 2007.

Irregular seismic data reconstruction based on exponential threshold model of POCS method

Irregular seismic data causes problems with multi-trace processing algorithms and degrades processing quality. We introduce the Projection onto Convex Sets （POCS） based image restoration method into the seismic data reconstruction field to interpolate irregularly missing traces. For entire dead traces, we transfer the POCS iteration reconstruction process from the time to frequency domain to save computational cost because forward and reverse Fourier time transforms are not needed. In each iteration, the selection threshold parameter is important for reconstruction efficiency. In this paper, we designed two types of threshold models to reconstruct irregularly missing seismic data. The experimental results show that an exponential threshold can greatly reduce iterations and improve reconstruction efficiency compared to a linear threshold for the same reconstruction result. We also analyze the anti- noise and anti-alias ability of the POCS reconstruction method. Finally, theoretical model tests and real data examples indicate that the proposed method is efficient and applicable.

Seismic data reconstruction based on low dimensional manifold model

Seismic data reconstruction is an essential and yet fundamental step in seismic data processing workflow,which is of profound significance to improve migration imaging quality,multiple suppression effect,and seismic inversion accuracy.Regularization methods play a central role in solving the underdetermined inverse problem of seismic data reconstruction.In this paper,a novel regularization approach is proposed,the low dimensional manifold model(LDMM),for reconstructing the missing seismic data.Our work relies on the fact that seismic patches always occupy a low dimensional manifold.Specifically,we exploit the dimension of the seismic patches manifold as a regularization term in the reconstruction problem,and reconstruct the missing seismic data by enforcing low dimensionality on this manifold.The crucial procedure of the proposed method is to solve the dimension of the patches manifold.Toward this,we adopt an efficient dimensionality calculation method based on low-rank approximation,which provides a reliable safeguard to enforce the constraints in the reconstruction process.Numerical experiments performed on synthetic and field seismic data demonstrate that,compared with the curvelet-based sparsity-promoting L1-norm minimization method and the multichannel singular spectrum analysis method,the proposed method obtains state-of-the-art reconstruction results.

Approximate Conditional Likelihood for Generalized Linear Models with General Missing Data Mechanism

The generalized linear model is an indispensable tool for analyzing non-Gaussian response data, with both canonical and non-canonical link functions comprehensively used. When missing values are present, many existing methods in the literature heavily depend on an unverifiable assumption of the missing data mechanism, and they fail when the assumption is violated. This paper proposes a missing data mechanism that is as generally applicable as possible, which includes both ignorable and nonignorable missing data cases, as well as both scenarios of missing values in response and covariate.Under this general missing data mechanism, the authors adopt an approximate conditional likelihood method to estimate unknown parameters. The authors rigorously establish the regularity conditions under which the unknown parameters are identifiable under the approximate conditional likelihood approach. For parameters that are identifiable, the authors prove the asymptotic normality of the estimators obtained by maximizing the approximate conditional likelihood. Some simulation studies are conducted to evaluate finite sample performance of the proposed estimators as well as estimators from some existing methods. Finally, the authors present a biomarker analysis in prostate cancer study to illustrate the proposed method.

The restricted EM algorithm under linear inequalities in a linear model with missing data

This paper discusses the maximum likelihood estimate of β under linear inequalities A0β≥ a in a linear model with missing data, proposes the restricted EM algo rithm and proves the convergence.

Data-driven fault diagnosis of control valve with missing data based on modeling and deep residual shrinkage network

A control valve is one of the most widely used machines in hydraulic systems.However,it often works in harsh environments and failure occurs from time to time.An intelligent and robust control valve fault diagnosis is therefore important for operation of the system.In this study,a fault diagnosis based on the mathematical model(MM)imputation and the modified deep residual shrinkage network(MDRSN)is proposed to solve the problem that data-driven models for control valves are susceptible to changing operating conditions and missing data.The multiple fault time-series samples of the control valve at different openings are collected for fault diagnosis to verify the effectiveness of the proposed method.The effects of the proposed method in missing data imputation and fault diagnosis are analyzed.Compared with random and k-nearest neighbor(KNN)imputation,the accuracies of MM-based imputation are improved by 17.87%and 21.18%,in the circumstances of a20.00%data missing rate at valve opening from 10%to 28%.Furthermore,the results show that the proposed MDRSN can maintain high fault diagnosis accuracy with missing data.

The AMSAA-BISE Model with Gap Intervals

In this paper, the AMSAA-BISE model with missing data is discussed. The ML estimates of model parameters and current MTBF are given, and the chi-squared test and a plot for cumulative number of failures versus cumulative testing time are used to test the goodness of fit for the model. This paper concludes with a numerical example to verify the model.

Using Boosted Regression Trees and Remotely Sensed Data to Drive Decision-Making

Challenges in Big Data analysis arise due to the way the data are recorded, maintained, processed and stored. We demonstrate that a hierarchical, multivariate, statistical machine learning algorithm, namely Boosted Regression Tree (BRT) can address Big Data challenges to drive decision making. The challenge of this study is lack of interoperability since the data, a collection of GIS shapefiles, remotely sensed imagery, and aggregated and interpolated spatio-temporal information, are stored in monolithic hardware components. For the modelling process, it was necessary to create one common input file. By merging the data sources together, a structured but noisy input file, showing inconsistencies and redundancies, was created. Here, it is shown that BRT can process different data granularities, heterogeneous data and missingness. In particular, BRT has the advantage of dealing with missing data by default by allowing a split on whether or not a value is missing as well as what the value is. Most importantly, the BRT offers a wide range of possibilities regarding the interpretation of results and variable selection is automatically performed by considering how frequently a variable is used to define a split in the tree. A comparison with two similar regression models (Random Forests and Least Absolute Shrinkage and Selection Operator, LASSO) shows that BRT outperforms these in this instance. BRT can also be a starting point for sophisticated hierarchical modelling in real world scenarios. For example, a single or ensemble approach of BRT could be tested with existing models in order to improve results for a wide range of data-driven decisions and applications.

基于GAN的软测量缺失数据生成方法研究

针对工业过程中传感器数据缺失造成软测量模型精度低的问题,提出一种基于生成对抗网络(generative adversarial nets,GAN)的传感器缺失数据生成方法。利用孤立森林算法检测出传感器数据的缺失区域;利用缺失数据属性特征训练条件生成对抗网络(conditional generative adversarial nets,CGAN),在CGAN的输入条件中添加随机序列作为附加信息迭代送入CGAN中生成数据,并借助WGAN-GP(wasserstein generative adversarial nets gradient penalty)成本函数提高网络训练的稳定性;针对缺失区域检测结果引入采样器,将采样的数据填补进缺失区域,形成完整数据集,以提高软测量模型精度。以镍闪速炉温度传感器数据为目标变量进行软测量建模,验证所提出的提高软测量模型精度方法的可行性与有效性。

空间自回归模型下不完整大数据缺失值插补算法

针对不完整大数据因其自身结构具有不规则性,导致在进行缺失值插补时计算量大、插补精度低的问题,提出空间自回归模型下不完整大数据缺失值插补算法。利用迁移学习算法在动态权重下过滤出原始数据中冗余数据,区分异常和正常数据,提取残缺数据,采用最小二乘回归对残缺数据实施修补。将缺失值插补分为3种类型,分别为一阶空间自回归模型插补、空间自回归模型插补和多重插补法。根据实际情况将修补后数据插补到合适的位置,实现不完整大数据缺失值插补。实验结果表明,所提方法具有良好的缺失值插补能力。

基于面板数据模型的拱坝缺失数据填补方法

混凝土拱坝作为重要的水工建筑物,由于监测设备故障、人为因素等影响,导致其监测数据频繁出现缺失的现象,降低了大坝安全评估与预测的有效性与准确性。传统方法多仅依赖单测点测值进行插补,忽略了测点之间的相关性与异质性。本文提出了一种基于面板数据模型的变形缺失数据插补方法。首先,改进传统变形相似性增量速度指标,解决了其分母可能等于零的问题。其次,提出了一种组合加权方法以计算变形相似性综合指标,并采用改进的基于密度聚类方法对变形监测点进行分类。随后,建立了面板模型,以填补不同区域内的缺失数据。本文提出的方法可以更准确地填补混凝土拱坝变形数据的缺失,从而能够有效地解决变形监测数据缺失的问题。

基于空间面板数据模型的拱坝变形缺失数据处理

变形监测信息缺失会对拱坝安全性态的分析造成困难甚至引起误判,需要采用科学合理的方法对缺失数据进行处理,从而获取完整可靠的监测数据。传统的缺失值处理方法仅考虑了拱坝测点的局部空间关联性,而拱坝变形整体性较强,在进行缺失值插补时有必要考虑拱坝所有测点间的时空关联性。鉴于此,从拱坝的整体性和测点的空间相关性出发,首先引入空间权重矩阵,证明拱坝变形具有显著的空间正自相关性;在此基础上,基于空间面板数据模型提出一种考虑拱坝整体时空关联性的变形缺失数据处理方法;最后结合某拱坝工程实例,验证了所构建模型的有效性。工程实例表明:该方法插补残差值低于SL 601—2013《混凝土坝安全监测技术规范》所规定的误差限值,具有较高的插补精度,可对变形监测缺失数据进行有效处理。

Variable Selection for Semiparametric Varying-Coefficient Partially Linear Models with Missing Response at Random

In this paper, we present a variable selection procedure by combining basis function approximations with penalized estimating equations for semiparametric varying-coefficient partially linear models with missing response at random. The proposed procedure simultaneously selects significant variables in parametric components and nonparametric components. With appropriate selection of the tuning parameters, we establish the consistency of the variable selection procedure and the convergence rate of the regularized estimators. A simulation study is undertaken to assess the finite sample performance of the proposed variable selection procedure.

基于电力营销大数据技术的反窃电检查应用分析

由于目前方法存在精度低和性能差的问题,提出基于电力营销大数据技术的反窃电检查应用分析方法。该方法采用因子分析模型对电力营销大数据进行降维处理,并利用缺失数据填补算法对降维处理后的缺失电力营销数据进行填补,对线损波动率、电流差异曲线和台区线损三者之间存在的关联进行分析,以此判断是否存在窃电行为,完成反窃电检查。实验结果表明,所提方法可准确检测到窃电行为发生的时间和次数,F-Measure值高,表明所提方法的检测精度高、性能好。

变系数部分非线性模型的分位数回归估计

研究纵向数据缺失下变系数部分非线性分位数回归模型的估计问题.利用逆概率加权法结合分位数回归给出参数估计和非参估计;在一定条件下,证明了所给估计量的渐近正态性;通过数值模拟,验证了所提方法的有效性.