基于最小哈希的网络单信道重复数据剔除算法

Duplicate Data Elimination of Network Single-Channel Based on Minimum Hash

剔除重复数据是保证网络高效运行不可缺少的步骤,但该过程易受信号强度、网络装置、路由器性能等问题的干扰。为此,提出基于最小哈希的网络单信道重复数据剔除算法。首先利用哈希算法中的散列函数对网络单信道数据实行聚类处理,然后采用带有监督判别的投影算法对聚类后的数据进行降维处理,最后采用代数签名预估数据,保证数据之间的计算开销最小,再构造最小哈希树生成校验值,在更新去重标签的同时,通过双层剔除机制完全剔除单信道中的重复数据。实验结果表明,该算法的执行时间短,且计算和存储开销较小。

Eliminating duplicate data is an indispensable step to ensure efficient network operation. But this process is susceptible to interference from signal strength, network device, router performance and other problems. Therefore, a minimum-hashing algorithm for single channel data elimination is proposed. First the hash function in the hash algorithm network is used for single channel data clustering, and then supervision discriminant projection algorithm is applied for clustering of data dimension reduction after processing, finally the algebraic sign estimate is used to guarantee the data between the computing cost minimum and to construct minimum hash tree generated calibration value, in the update to heavy tags. The repeated data in a single channel is completely eliminated by double-layer culling mechanism. Experimental results show that the algorithm has short execution time and low computation and storage cost.