基于BERT-BiLSTM模型的短文本自动评分系统

Short text automatic scoring system based on BERT-BiLSTM model

针对短文本自动评分中存在的特征稀疏、一词多义及上下文关联信息少等问题,提出一种基于BERT-BiLSTM(bidirectional encoder representations from transformers-bidirectional long short-term memory)的短文本自动评分模型.使用BERT(bidirectional encoder representations from transformers)语言模型预训练大规模语料库习得通用语言的语义特征,通过预训练好的BERT语言模型预微调下游具体任务的短文本数据集习得短文本的语义特征和关键词特定含义,再通过BiLSTM(bidirectional long short-term memory)捕获深层次上下文关联信息,最后将获得的特征向量输入Softmax回归模型进行自动评分.实验结果表明,对比CNN(convolutional neural networks)、CharCNN(character-level CNN)、LSTM(long short-term memory)和BERT等基准模型,基于BERT-BiLSTM的短文本自动评分模型所获的二次加权kappa系数平均值最优.

Aiming at the problems of sparse features,polysemy of one word and less context related information in short text automatic scoring,a short text automatic scoring model based on bidirectional encoder representations from transformers-bidirectional long short-term memory(BERT-BiLSTM)is proposed.Firstly,the large-scale corpus is pre-trained with bidirectional encoder representations from transformers(BERT)language model to acquire the semantic features of the general language.Then the semantic features of short text and the semantics of keywords in a specific context are acquired through the short text data for the pre-fine tuning downstream specific tasks set prefined by BERT.And then the deep-seated context dependency is captured through bidirectional long short-term memory(BiLSTM).Finally,the obtained feature vectors are input into Softmax regression model for automatic scoring.The experimental results show that compared with other benchmark models of convolutional neural networks(CNN),character-level CNN(CharCNN),long short-term memory(LSTM)and BERT,the short text automatic scoring model based on BERT-BiLSTM achieves the best average value of quadratic weighted kappa coefficient.