Nowadays a common problem when processing data sets with the large number of covariates compared to small sample sizes (fat data sets) is to estimate the parameters associated with each covariate. When the number of c...Nowadays a common problem when processing data sets with the large number of covariates compared to small sample sizes (fat data sets) is to estimate the parameters associated with each covariate. When the number of covariates far exceeds the number of samples, the parameter estimation becomes very difficult. Researchers in many fields such as text categorization deal with the burden of finding and estimating important covariates without overfitting the model. In this study, we developed a Sparse Probit Bayesian Model (SPBM) based on Gibbs sampling which utilizes double exponentials prior to induce shrinkage and reduce the number of covariates in the model. The method was evaluated using ten domains such as mathematics, the corpuses of which were downloaded from Wikipedia. From the downloaded corpuses, we created the TFIDF matrix corresponding to all domains and divided the whole data set randomly into training and testing groups of size 300. To make the model more robust we performed 50 re-samplings on selection of training and test groups. The model was implemented in R and the Gibbs sampler ran for 60 k iterations and the first 20 k was discarded as burn in. We performed classification on training and test groups by calculating P (yi = 1) and according to [1] [2] the threshold of 0.5 was used as decision rule. Our model’s performance was compared to Support Vector Machines (SVM) using average sensitivity and specificity across 50 runs. The SPBM achieved high classification accuracy and outperformed SVM in almost all domains analyzed.展开更多
One paper in a preceding issue of this journal has introduced the Bayesian Ying-Yang(BYY)harmony learning from a perspective of problem solving,parameter learning,and model selection.In a complementary role,the paper ...One paper in a preceding issue of this journal has introduced the Bayesian Ying-Yang(BYY)harmony learning from a perspective of problem solving,parameter learning,and model selection.In a complementary role,the paper provides further insights from another perspective that a co-dimensional matrix pair(shortly co-dim matrix pair)forms a building unit and a hierarchy of such building units sets up the BYY system.The BYY harmony learning is re-examined via exploring the nature of a co-dim matrix pair,which leads to improved learning performance with refined model selection criteria and a modified mechanism that coordinates automatic model selection and sparse learning.Besides updating typical algorithms of factor analysis(FA),binary FA(BFA),binary matrix factorization(BMF),and nonnegative matrix factorization(NMF)to share such a mechanism,we are also led to(a)a new parametrization that embeds a de-noise nature to Gaussian mixture and local FA(LFA);(b)an alternative formulation of graph Laplacian based linear manifold learning;(c)a codecomposition of data and covariance for learning regularization and data integration;and(d)a co-dim matrix pair based generalization of temporal FA and state space model.Moreover,with help of a co-dim matrix pair in Hadamard product,we are led to a semi-supervised formation for regression analysis and a semi-blind learning formation for temporal FA and state space model.Furthermore,we address that these advances provide with new tools for network biology studies,including learning transcriptional regulatory,Protein-Protein Interaction network alignment,and network integration.展开更多
In this work,we have proposed a generative model,called VAE-KRnet,for density estimation or approximation,which combines the canonical variational autoencoder(VAE)with our recently developed flow-based generativemodel...In this work,we have proposed a generative model,called VAE-KRnet,for density estimation or approximation,which combines the canonical variational autoencoder(VAE)with our recently developed flow-based generativemodel,called KRnet.VAE is used as a dimension reduction technique to capture the latent space,and KRnet is used to model the distribution of the latent variable.Using a linear model between the data and the latent variable,we show that VAE-KRnet can be more effective and robust than the canonical VAE.VAE-KRnet can be used as a density model to approximate either data distribution or an arbitrary probability density function(PDF)known up to a constant.VAE-KRnet is flexible in terms of dimensionality.When the number of dimensions is relatively small,KRnet can effectively approximate the distribution in terms of the original random variable.For high-dimensional cases,we may use VAE-KRnet to incorporate dimension reduction.One important application of VAE-KRnet is the variational Bayes for the approximation of the posterior distribution.The variational Bayes approaches are usually based on the minimization of the Kullback-Leibler(KL)divergence between the model and the posterior.For highdimensional distributions,it is very challenging to construct an accurate densitymodel due to the curse of dimensionality,where extra assumptions are often introduced for efficiency.For instance,the classical mean-field approach assumes mutual independence between dimensions,which often yields an underestimated variance due to oversimplification.To alleviate this issue,we include into the loss the maximization of the mutual information between the latent random variable and the original random variable,which helps keep more information from the region of low density such that the estimation of variance is improved.Numerical experiments have been presented to demonstrate the effectiveness of our model.展开更多
文摘Nowadays a common problem when processing data sets with the large number of covariates compared to small sample sizes (fat data sets) is to estimate the parameters associated with each covariate. When the number of covariates far exceeds the number of samples, the parameter estimation becomes very difficult. Researchers in many fields such as text categorization deal with the burden of finding and estimating important covariates without overfitting the model. In this study, we developed a Sparse Probit Bayesian Model (SPBM) based on Gibbs sampling which utilizes double exponentials prior to induce shrinkage and reduce the number of covariates in the model. The method was evaluated using ten domains such as mathematics, the corpuses of which were downloaded from Wikipedia. From the downloaded corpuses, we created the TFIDF matrix corresponding to all domains and divided the whole data set randomly into training and testing groups of size 300. To make the model more robust we performed 50 re-samplings on selection of training and test groups. The model was implemented in R and the Gibbs sampler ran for 60 k iterations and the first 20 k was discarded as burn in. We performed classification on training and test groups by calculating P (yi = 1) and according to [1] [2] the threshold of 0.5 was used as decision rule. Our model’s performance was compared to Support Vector Machines (SVM) using average sensitivity and specificity across 50 runs. The SPBM achieved high classification accuracy and outperformed SVM in almost all domains analyzed.
基金supported by the General Research Fund from Research Grant Council of Hong Kong(Project No.CUHK4180/10E)the National Basic Research Program of China(973 Program)(No.2009CB825404).
文摘One paper in a preceding issue of this journal has introduced the Bayesian Ying-Yang(BYY)harmony learning from a perspective of problem solving,parameter learning,and model selection.In a complementary role,the paper provides further insights from another perspective that a co-dimensional matrix pair(shortly co-dim matrix pair)forms a building unit and a hierarchy of such building units sets up the BYY system.The BYY harmony learning is re-examined via exploring the nature of a co-dim matrix pair,which leads to improved learning performance with refined model selection criteria and a modified mechanism that coordinates automatic model selection and sparse learning.Besides updating typical algorithms of factor analysis(FA),binary FA(BFA),binary matrix factorization(BMF),and nonnegative matrix factorization(NMF)to share such a mechanism,we are also led to(a)a new parametrization that embeds a de-noise nature to Gaussian mixture and local FA(LFA);(b)an alternative formulation of graph Laplacian based linear manifold learning;(c)a codecomposition of data and covariance for learning regularization and data integration;and(d)a co-dim matrix pair based generalization of temporal FA and state space model.Moreover,with help of a co-dim matrix pair in Hadamard product,we are led to a semi-supervised formation for regression analysis and a semi-blind learning formation for temporal FA and state space model.Furthermore,we address that these advances provide with new tools for network biology studies,including learning transcriptional regulatory,Protein-Protein Interaction network alignment,and network integration.
基金X.Wan has been supported by NSF grant DMS-1913163S.Wei has been supported by NSF grant ECCS-1642991.
文摘In this work,we have proposed a generative model,called VAE-KRnet,for density estimation or approximation,which combines the canonical variational autoencoder(VAE)with our recently developed flow-based generativemodel,called KRnet.VAE is used as a dimension reduction technique to capture the latent space,and KRnet is used to model the distribution of the latent variable.Using a linear model between the data and the latent variable,we show that VAE-KRnet can be more effective and robust than the canonical VAE.VAE-KRnet can be used as a density model to approximate either data distribution or an arbitrary probability density function(PDF)known up to a constant.VAE-KRnet is flexible in terms of dimensionality.When the number of dimensions is relatively small,KRnet can effectively approximate the distribution in terms of the original random variable.For high-dimensional cases,we may use VAE-KRnet to incorporate dimension reduction.One important application of VAE-KRnet is the variational Bayes for the approximation of the posterior distribution.The variational Bayes approaches are usually based on the minimization of the Kullback-Leibler(KL)divergence between the model and the posterior.For highdimensional distributions,it is very challenging to construct an accurate densitymodel due to the curse of dimensionality,where extra assumptions are often introduced for efficiency.For instance,the classical mean-field approach assumes mutual independence between dimensions,which often yields an underestimated variance due to oversimplification.To alleviate this issue,we include into the loss the maximization of the mutual information between the latent random variable and the original random variable,which helps keep more information from the region of low density such that the estimation of variance is improved.Numerical experiments have been presented to demonstrate the effectiveness of our model.
