摘要
数据降维是从高维数据中挖掘有效信息的必要步骤。传统的主成分分析(PCA)算法应用于超高维稀疏数据降维时,存在着无法将所有数据特征一次性读入内存以进行分析计算的问题,而之后提出的分块处理PCA算法由于耗时太长,并不能满足实际需求。本文引入信息熵的思想对PCA算法进行改进,提出E-PCA算法,先利用信息熵对数据进行特征筛选,剔除大部分无用特征,再使用PCA算法对处理后的超高维稀疏数据进行降维。通过实验结果表明,在保留相同比例原数据信息的情况下,本文提出的基于信息熵的E-PCA算法在内存占用、运行时间以及降维结果都优于分块处理PCA算法。
Data dimensionality reduction is a necessary step in mining effective information fromhigh-dimensional data. When applying the traditional principal component analysis (PCA) algorithm tohigh-dimensional sparse data dimensionality reduction, there is a problem that unable to read all data features atonce into memory for analysis and calculation, furthermore, the improved block processing PCA algorithm also cannot meet the actual requirements because of the time consuming. In this paper, we propose the E-PCA algorithm byintroducing the concept of information entropy to improve the PCA algorithm. First, the useless features areeliminated through feature selection based on information entropy, and then PCA algorithm is used to reduce thedimensionality of large, high-dimensional sparse data. The experimental results show that in the case of keeping thesame proportion of raw data, the information entropy-based E-PCA algorithm proposed in this paper is superior toblock processing PCA algorithm in terms of memory usage, run time and the results of dimension reduction.
出处
《电子科技大学学报》
EI
CAS
CSCD
北大核心
2018年第2期235-241,共7页
Journal of University of Electronic Science and Technology of China
基金
国家自然科学基金(61472064
61602096)
四川省科技计划项目(2016FZ0002
2015JY0178
2016ZC2575)
四川省教育厅重点项目(17ZA0322)
中央高校基本科研基金(ZYGX2014J051
ZYGX2014J066)
网络与数据安全四川省重点实验室开放课题(NDSMS201606)
关键词
分块处理
降维处理
高维稀疏大数据
信息熵
主成分分析
block processing
dimensionality reduction
high-dimensional sparse data
informationentropy
principal component analysis