Web大数据系统数据源选择

Data Source Selection for Web Big Data System

如何从数量众多的Web数据源集合中选择数量合适的数据源,使得在满足特定查询需求的前提下尽可能地减少访问数据源的数量,是Web大数据系统集成中的关键问题之一。提出了一个两阶段数据源选择方案:第一阶段通过各个数据源模式与中间模式的相似度选择与查询相关度高的数据源,通过计算依赖数据源的质量来选取质量较好的数据源;第二阶段基于最大熵理论计算数据源之间的重复率,设计实现了一个查询最小代价模型动态选择数据源算法。最后在实验平台上对算法进行了评估,实验表明该算法具有较高的效率与扩展性。

How to select the appropriate data source from the large number of Web data sources,so as to reduce the number of accessing data sources,is one of the key issues in the integration of Web big data system.This paper proposes a two-stage data source selection method.The first stage is to select the data source with the high similarity to the middle schema and select the data source with the high reliability by computing the quality of dependent data source.In the second stage,a time-cost minimization query algorithm is designed for source permutation.To calculate the repetition rate of the data source,the maximum entropy theory is applied in the algorithm.Finally,the algorithmis evaluated on the experimental platform.The experiments show that the proposed algorithm has high efficiency and scalability compared with other algorithms.