基于Hadoop的并行化命名实体识别模型研究

Research on parallel named entity recognition model based on Hadoop

针对现有命名实体识别存在数据处理效率低的问题,该文提出了一种并行化Block-BAC模型。提出了前处理中的数据分块优化算法,并基于Hadoop实现并行化的运作机制;采用局部注意力优化机制,有效减少模型的隐层节点。与已有的BERT-BAC模型相比,在确保较高F;值(精确率和召回率的调和平均数)的情况下,该模型训练时间和实体识别时间分别缩短60.36%、39.43%,具有更广泛的实用性。

A parallel Block-BAC model is proposed to solve the problem of inefficient data processing in the current named entity recognition in this paper. The data block optimization algorithm in preprocessing is proposed,and the parallel operation mechanism is realized based on Hadoop. Local attention optimization mechanism is adopted to reduce the hidden nodes of the model. Compared with the existing BERT-BAC model, the training time and the entity recognition time of the model are reduced by 60.36% and 39.43% respectively, with a higher F;(harmonic average of accuracy rate and recall rate), which has wider practicability.