基于CNN的安防数据相似重复记录检测模型

APPROXIMATELY DUPLICATE RECORD DETECTION MODEL FOR SECURITY DATA BASED ON CNN

安防行业的结构化数据中存在大量的相似重复记录,传统的相似重复记录检测算法的识别率很难满足安防行业的实际需求。针对这种情况,引入了卷积神经网络模型,设计两种以LeNet-5模型为基础的改进模型,一种是输入为词向量矩阵的模型,另一种是输入为相似度矩阵的模型。实验表明,输入为词向量矩阵的模型的精确率和召回率均达到了96%以上,输入为相似度矩阵的模型的精确率和召回率高达98%,并且K折交叉验证的结果说明模型具有较强的泛化能力。

There are a lot of approximately duplicate record in the structured data of security industry.The recognition rate of traditional approximately duplicate record detection algorithm is difficult to meet the actual demand of security industry.In order to solve the above problems,a convolutional neural network model was introduced and two improved models based on LeNet-5 model were designed.One was the model with input as word embedding matrix,the other is the model with input as similarity matrix.The experiments show that the precision rate and recall rate of the model with input as word embedding matrix reach more than 96%.And the precision rate and recall rate of the model with input as a similarity matrix reach up to 98%.The experimental results of K-fold cross validation show that both models have strong generalization ability.