基于Web的重复属性自动识别方法

Automatic Web-based duplicate attribute resolution method

在建立数据仓库的过程中,需要从多个数据源导入数据。这些数据存在大量相似重复记录,严重影响了数据利用率和决策质量。因此,相似重复记录的检测已经成为数据仓库等领域的热点研究问题,而重复属性的识别是完成相似重复记录检测的关键。提出一种高效的基于Web的重复属性自动识别算法,该算法使用搜索引擎返回的摘要和URL信息计算属性相似度,并使用查询探针提高查询准确度。实验结果表明该算法有较高的查全率。

It needs to import data from different sources when building a data warehouse, which results in the increase of approximately duplicated records, adversely affecting the data utilization and the quality of making decisions. Therefore, detecting approximately duplicated records has become an intensive research subject. Duplicate attribute resolution is the key of approximately duplicated records detecting. This paper proposes an efficient, Web-based algorithm to automatically recognize duplicated attributes. This novel algorithm uses snippets and URL information returned by search engine to calcu-late attribute similarity, and further improves recognition precision by inserting query probes. The results show that the algo-rithm improves recall.