人在回路的数据融合系统

Human-in-the-Loop Data Integration System

一个端到端的数据融合系统在如下几个阶段需要用户的反馈,包括为实体匹配收集训练数据的阶段,对计算出来的实体聚类进行纠错的阶段以及对每个实体聚类中的数据形式进行转换的阶段.经过这些步骤之后,数据融合系统可以将每个实体分组中的实体合并成一条权威的实体表示(被称作“黄金记录”).传统的数据融合系统通常会按照特定的顺序,在用户时间代价不超过预算的前提下,依次询问特定集合中的问题并且获得用户反馈.可是,由于这些问题是高度相关的,一个问题的答案会影响到数据处理管道中任何一个阶段的结果,所以将各个阶段交替进行并且对他们的执行顺序进行全局优化能带来显著的收益.在这篇文章中,我们提出一个“人在回路”的数据融合系统,通过交替询问不同种类的问题来减少获得用户回答的代价.基于这个框架,我们提出了一些不同种类问题能够带来的数据融合质量提升的收益评价模型,以及不同问题需要的用户回答时间的代价评估模型.在这个端到端的“人在回路”数据融合系统中,我们开发出一个问题调度框架来精心选择问题集合以及问题询问的顺序,最终实现利用同等的用户代价让实体合并的精确度从基线方法的70%提升到90%.

An end-to-end data integration system requires human feedback in several phases,including collecting training data for entity matching,debugging the resulting clusters,confirming transformations applied on these clusters for data standardization,and finally,reducing each cluster to a single,canonical representation(or“golden record”).The traditional wisdom is to sequentially apply the human feedback,obtained by asking specific questions,within some budget in each phase.However,these questions are highly correlated;the answer to one can influence the outcome of any of the phases of the pipeline.Hence,interleaving them has the potential to offer significant benefits.In this paper,we propose a human-in-the-loop framework that interleaves different types of questions to optimize human involvement.We propose benefit models to measure the quality improvement from asking a question,and cost models to measure the human time it takes to answer a question.We develop a question scheduling framework that judiciously selects questions to maximize the accuracy of the final golden records.Experimental results on three real-world datasets show that our holistic method significantly improves the quality of golden records from 70%to 90%,compared with the state-of-the-art approaches.