大数据环境下的电子商务商品实体同一性识别

Recognizing the Same Commodity Entities in Big Data

怎样从多源异构的、自治独立的、多样化的、不一致的电子商务数据中找出同一商品实体是当前面临的主要挑战.通过分析不同平台的数据特征,首先建立基于商品属性?值的索引模型,构造商品属性-值的全局模式图并进行模式集成,形成模式统一、质量高效的商品信息数据;而后基于层次概率模型对商品的同一性进行多层相似度量;最终完成商品实体识别,并归一化输出满足同一性的商品集和关联属性并进行排序.基于Hadoop平台对3个B2C电子商务数据源中的商品进行了实验,并与传统方法和产品进行了比较,实验结果证明了本框架的可行性、精确性和高效性.

The recent blossom of big data and e-commerce has revolutionized our life by providing everyone with the ease and fun never before. How to identify the same commodity entities from these multi-source heterogeneous, fragmented, various and inconsistent e-commerce data for better business intelligence raises a very valuable and challenging topic. In this light, we analyze the characteristics of Web big data and collect the crawled original commodity information data from the different e- commerce platforms, which are the multi-source heterogeneous and mass scales of data. Then, we build an index model based on commodity＇s attributes and values, and construct a global model map to record the commodity＇s attribute and value, and form the unified model and high efficient commodity information for the next step. And we measure the similarity of the commodity＇s identity on the multilayer hierarchical probabilistic model, including identifying the possible candidate commodity set, similarity filtering the candidate commodity set and similarity filtering based on the special items of candidate commodities set. Finally, we output We also evaluate our method on the datasets collecte platforms with Hadoop framework. Experimental method. the same commodity set in the inverted index list. d from Chinese three main-stream B2C e-commerce results show the accuracy and effectiveness of our