基于ALBERT预训练模型生成式文本摘要

Pretraining based on ALBERT model for abstract text summarization

针对如何生成更高质量的文本摘要提出一种基于预训练的编码器-解码器框架。该框架可以根据输入序列分两阶段生成输出序列。对于模型的编码器,使用ALBERT将输入序列编码为上下文表示。解码器在第一阶段使用一个基于Transformer的解码器来生成一个草稿输出序列;第二阶段对草稿序列中每个词进行掩码,并将其输入到ALBERT中,然后通过结合输入序列和ALBERT生成的草稿表示,使用基于Transformer的解码器来预测每一个掩码位置的词。在ROUGE上评估了此模型。模型生成的摘要解决了生成摘要不自然的问题,并在LCSTS上取得了很好的效果。

An encoder-decoder framework based on pretraining is proposed to generate higher quality text summaries.The framework can generate output sequence from input sequence in two stages.For the encoder of the model,use ALBERT to encode the input sequence as a context representation.For the decoder,in the first phase,a Transformer based decoder is used to generate a draft output sequence.In the second phase,each word in the draft sequence is masked and entered into ALBERT,then the words at each mask position are predicted using a Transformer based decoder by combining the input sequence with the draft representation generated by ALBERT.This model was evaluated on ROUGE.Abstract generated by model solves the problem of unnatural abstract generation and achieves good results in LCSTS.