数据源敏感的多源XML数据相似度量方法

Similarity Measure of Multi-Source XML Data by Means of Data Source-Sensitivity

将预处理后的XML数据当作文本信息采用词频-逆向文档频率(TF-IDF)模型进行处理时,逆向文档频率作为词项权重有其不足之处.为此,文中定义了词项的数据源敏感度作为逆向文档频率(IDF)的修正系数.其值取决于提供此词项的数据来源于不同数据源的概率,概率大则其值大,反之则其值小.然后在修正后的词项权重向量的基础上,定义了相似度函数.最后在模拟、真实数据集上进行数据重复检测实验.结果表明,新方法获得了更高的F测度值.这说明考虑词项的数据源敏感度可提高相似度函数的有效性.

When preprocessed XML data are used as text information to be dealt with by the TF-IDF （ Term Fre-quency-Inverse Document Frequency ） model, the IDF as the weight of terms has imperfection of its own .In order to solve this problem , the data source-sensitivity of terms is defined as the modification coefficient of the IDF .Its value depends on the probability which provides the term with the data from different sources .When the probability is big, its value is big, and vice versa.Then, the similarity function is defined on the basis of the weight vector of the fixed terms.Finally, experiments of detecting duplicate XML data from multiple sources are conducted on real and simulated datasets .The results show that the proposed method achieves a higher F measure value , which indi-cates that the data source-sensitivity of terms helps improve the effectiveness of similarity function .