引入位置编码机制对抗网络的文本生成模型

Generative Adversarial Networks with Position Encoding for Text Generation

针对目前对抗网络文本生成模型在生成文本时,出现词与词之间位置关系紊乱导致文本逻辑不通的问题,该文提出了一种引入位置编码机制对抗网络的文本生成模型(Position-Encoding GAN,PE_GAN)并进行探讨和验证。在对抗神经网络模型的基础上引入位置编码机制,可以通过带有位置编码的词向量来标记文本中词与词之间的位置关系,生成器和判别器使用GRU神经网络的门控机制来减少梯度消失,同时利用蒙特卡洛策略思想来降低数据过拟合风险并提高生成文本的准确性。为了验证PE_GAN模型的有效性,使用开源数据和网络爬取的小说和新闻文本共同作为实验的数据集,结果表明:该模型中生成器和判别器loss值的差距比对比模型小,表明生成的文本更加接近真实文本;与Gumbel-softmax GAN模型、seq-GAN模型和LFMGAN模型相比,PE_GAN模型的BLEU-2、BLEU-3和BLEU-4的值分别都有明显的提高,表明引入位置编码机制后可以改善生成文本的逻辑性,由此可知该模型有较好的应用性。

For the problem that text logic is not logical due to the disorder of the position relationship between words when text is generated in the current adversarial network text generation model,we propose a text generation model(Position-Encoding GAN,PE_GAN)which introduces a position encoding mechanism to adversarial network,and discusses and validates it.By introducing positional encoding to the adversarial neural network model,the position relationship between words in the text is marked using word vectors with positional encoding.The generator and discriminator utilize the gate mechanism of the GRU neural network to alleviate gradient vanishing,while employing the Monte Carlo strategy to reduce the risk of overfitting and improve the accuracy of generated text.To verify the effectiveness of the PE_GAN,open source data and text from novels and news articles obtained through web scraping area used as the experimental dataset.It is showed that the difference in loss values between generator and discriminator in this model is smaller than that of the comparison models,indicating the generated text is closer to real text.In comparison to the Gumbel-softmax GAN,seq-GAN and LFMGAN,the PE_GAN shows a signficant improvement in BLEU-2,BLEU-3 and BLEU-4 values.This suggests that introducing positional encoding mechanism can improve the logical coherence of the generated text,indicating that the model has good applicability.