一种基于Simhash算法的重复域名数据去重方法

Method for deleting duplicate domain name data based on Simhash algorithm

随着数字科学技术的发展,各领域需要传输和存储的数据量急剧上升。然而传输和存储的数据中重复数量占据了很大的比例,这不仅会增加使用数据的成本,也会影响处理数据的效率。域名是一种存储量大而且对处理速率有极高要求的数据,为了节约域名解析系统的存储成本,提高传输效率,本文在原有数据去重技术的基础上,引入了Simhash算法,结合域名数据的结构特征,改进数据分词和指纹值计算方式,提出了一种基于Simhash算法的重复域名数据去重方法。实验结果表明,相比于传统的数据去重技术,该方法对删除重复域名数据效率更高,具有较好的实际应用价值。

With the development of digital science and technology,the amount of data that needs to be transmitted and stored in various fields has risen sharply.However,the number of repetitions in these data occupies a large proportion.This not only increases the cost of using data,but also reduces the efficiency of data processing.Domain name is a kind of data with large storage capacity and extremely high requirements for processing speed.In order to save storage cost and improve transmission efficiency,this paper proposes a method for deleting duplicate domain name data based on Simhash algorithm.Compared with the traditional data deduplication technology,this method combines the structural characteristics of the domain name data,and introduces the Simhash algorithm to design a deduplication method for the domain name data.The experimental results show that compared with the traditional data deduplication technology,this method is more efficient in deleting duplicate domain name data and has better practical application value.