NKCorpus:利用海量网络数据构建大型高质量中文数据集

NKCorpus: Extracting High Quality Large Chinese Dataset from Web Data

【目的】大规模、高质量的中文数据集对于大型中文预训练语言模型及其他自然语言处理模型的训练至关重要,因此需要设计并完善一种可以构建大规模中文数据集的框架。【方法】利用语言提取、文本清洗、数据去重等多种方法对原始数据进行处理获取数据集,并利用并行技术对数据处理框架的效率进行优化。【结果】提出了一个流程完善且高效的可以利用海量网络数据构建大型高质量中文数据集的框架NKCorpus,并且利用NKCorpus构建了约700GB的可直接用于中文预训练语言模型的训练工作的高质量中文数据集。【结论】NKCorpus已能够基本满足当前对于大规模、高质量中文数据集的高效构建需求。

[Objective]For large-scale Chinese pre-trained language models or other natural language processing models,it is very important to collect and process large-scale high-quality Chinese data for model training.Therefore,a comprehensive large-scale dataset construction framework is required.[Methods]We use pipeline preprocessing procedures such as language extraction,text cleaning,and deduplication to process data.The performance of our framework is also optimized by parallel computing techniques.[Results]A comprehensive and efficient dataset construction framework NKCorpus is proposed to construct large-scale high-quality Chinese corpus datasets from massive web data,and a high-quality Chinese dataset of about 700GB is constructed using NKCorpus.[Conclusions]NKCorpus can meet the current needs for the efficient construction of large-scale,high-quality Chinese datasets.