基于关联规则的海量重复数据消除系统设计

Design of massive repeating data elimination system based on association rules

结合传统消除系统中重复数据吞吐量不可控、相似判断完成时间长的问题,利用关联规则,改进设计了海量重复数据消除系统。为增强相似数据搜索速度、保证消除可靠性,对系统的总体框架进行设计,依据框架将硬件部分重新划分为重复数据检测、总吞吐量提升、消除可靠性保证、系统安全四个主要模块。应用TCP/IP,NetBEUI,IPX/SPX三种协议相结合的方式代替传统系统的NetBEUI传输协议作为核心运行基础,建立基础数据序列关联规则优化重复数据检测编码,确保海量重复数据消除流程的高效性和稳定性,完成消除系统设计。采用传统系统和改进系统对比的方式进行实验,实验结果表明,该系统的重复数据吞吐量始终维持在可控范围内,且相似判断完成时间可缩短至传统系统的1 2左右。

The traditional data elimination system has the disadvantages of uncontrollable repeating data throughput and long similar judgment completion time.Therefore,a massive repeating data elimination system based on association rules is designed.The system overall framework is designed to enhance the search speed of similar data and guarantee the elimination of reliability.The hardware of the system is divided into repeated data detection module,total throughput upgrading module,reliability assurance module and system security module.The combination mode of TCP/IP,NetBEUI and IPX/SPX is used to replace the traditional NetBEUI transmission protocol as the core operation basis.The basic data sequence association rules are established to optimize the repeated data detection coding,so as to ensure the efficiency and stability of the massive duplication data elimination process,and complete the system design.The contrast experimental results show that the massive repeating data elimination system based on association rules can maintain the repeating data throughput in the controllable range,and shorten the similar judgment completion time to about half of the traditional system.