基于输入样本和主数据的编辑规则挖掘算法

Method for Discovering Editing Rules From Sample Inputs and Master Data

基于编辑规则和主数据的数据修复技术能自动地、确切地修复不一致数据,但目前编辑规则的获取主要依靠专业人员的定义.为了实现数据清洗全自动化,数据规则的挖掘技术近年来成为研究热点,针对条件函数依赖提出的挖掘算法主要有CFDMiner,CTANE,FastCFD.在此基础上,扩展条件函数依赖(CFD)的定义,在编辑规则的定义下提出了一种基于输入样本和主数据的编辑规则挖掘算法,主要思路是从输入样本中挖掘出CFD,然后根据输入样本与主数据在属性上的定义域相似性求出输入样本在主数据中的对应属性,从而形成带模式组的编辑规则,此算法能有效地挖掘编辑规则.且所挖掘的编辑规则按照编辑规则语义能有效地进行数据修复.

Data repairing based on editing rules and master data can automatically and exactly fix inconsistent data, but editing rules mainly relies on the definition by professional staff at present. To achieve data cleaning automatically in the whole process, the techniques for discovering data rules become a hot research topic in recent years. The algorithms for mining CFDs mainly involve CFDMiner, CTANE, FastCFD. Based on the above techniques, we provide a mining algorithm for editing rule, which is based on sample inputs and master data under the extension definition of CFD and the definition of edit rules. The main ideas is as below： Mining CFD from sample inputs firstly; then according to the domain similarity between input samples and master data, we can get the corresponding properties of input samples from the master data, forming editing rules with pattern group. The algorithm can effectively discover edit rules. And the mined edit rules can effectively repair the data in accordance with the semantic of the rules.