对话生成受到学术界和工业界的广泛关注,然而现有的对话生成模型忽视了源句与目标句在主题上的相关性,使得生成的内容质量大打折扣。该文提出了一个基于显式主题增强的单轮对话生成模型,首先通过多种方式微调BERT模型得到高质量的词向量,然后利用LDA模型提取源句与目标句中的主题词作为额外信息输入,通过主题过滤和文本过滤两个模块,对输入信息进行筛选,从而显式地增强主题相关性,最后利用变分自动编码器独特的重构再采样特性生成语义一致、主题相关且内容丰富的回复。该文在NLPCC2017 发布的ECG数据集上进行相关实验,实验结果表明,该文方法与主流的基于Seq2Seq模型的方法相比,在生成回复的流畅性和主题相关性等方面均有明显提升。
Abstract
Dialogue generation has always been widely concerned by industry and academia. To capture the relevance between the source sentence and the target sentence on the topic, this paper proposes a single round dialogue generation model based on explicit topic enhancement. In this model, we design a variety of ways to fine tune the Bert model to get high-quality word vectors. We use LDA topic model to extract topic words for sentence pairs as additional input to explicitly enhance the topic relevance. Then, through the unique refactoring and resampling feature of the variational auto-encoder, we can generate semantically consistent, topic related and content rich replies. Experiments on ECG data set of NLPCC2017 show that the proposed method is better than the current mainstream seq2seq model in terms of sentence fluency and topic relevance.
关键词
对话生成 /
单轮对话 /
主题性 /
语义一致 /
信息筛选
{{custom_keyword}} /
Key words
dialogue generation /
single round dialogue /
thematic /
semantic consistency /
information filtering
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks[C]//Proceedings of the Advances in Neural Information Processing Systems,2014: 3104-3112.
[2] BOWMAN S R, VILNIS L, VINYALS O, et al. Generating sentences from a continuous space[J]. arXiv:1511.06349v4,2016.
[3] VINYALS O, LE Q. A neural conversational model[C]//Proceedings of the 31st International Conference on Machine Learning, 2015.
[4] SHANG L, LU Z, HANG L. Neural responding machine for short-text conversation[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 2015: 1577-1586.
[5] MO K, LI S, ZHANG Y, et al. Personalizing a dialogue system with transfer reinforcement learning[J]. arXiv Preprint arXiv: 1610.02891, 2016.
[6] LI J, MONROE W, DAN J. A simple, fast diverse decoding algorithm for neural generation[J]. arXiv Preprint arXiv:1611.08562, 2016.
[7] SHEN X, HUI S, LI Y, et al. A conditional variational framework for dialog generation[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017: 504-509.
[8] WANG H, LU Z, LI H, et al. A dataset for research on short-text conversation[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2013: 935-945.
[9] MIKOLOV T, KARAFIT M, BURGET L, et al. Recurrent neural network based language model[C]//Proceedings of the Annual Conference of the International Speech Communication Association, 2010.
[10] ZHAO W X, JING J, WENG J, et al. Comparing twitter and traditional media using topic models[C]//Proceedings of the European Conference on Advances in Information Retrieval, 2011: 338-349.
[11] SORDONI A, et al, A neural network approach to context-sensitive generation of conversational responses[C]//Proceedings of the Association for Computational Linguistics, 2015:196-205.
[12] XING C, WU W, WU Y, et al. Topic aware neural response generation[C]//Proceedings of the 31st AAAI Conference on Artificial Intelligence, 2017.
[13] 王豪,郭斌,郝少阳,等. 基于深度学习的个性化对话内容生成方法[J]. 图学学报, 2020,041(002): 210-216.
[14] SERBAN I V, SORDONI A, BENGIO Y, et al. Building end-to-end dialogue systems using generative hierarchical neural network models[C]//Proceedings of the 30th AAAI Conference on Artificial Intelligence, 2015: 3776-3783.
[15] HOCHREITER S, SCHMIDHUBER J. Long short-term memory[J]. Neural Computation, 1997,9(8): 1735-1780.
[16] VASWANI, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6000-6010.
[17] BAO S H, HUANG W, et al. PLATO: pre-trained dialogue generation model with discrete latent variable[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 85-96.
[18] DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 4171-4186.
[19] MOU L, SONG Y, YAN R, et al. Sequence to backward and forward sequences: A content-introducing approach to generative short-text conversation[C]//Proceedings of the COLING, 2016: 3349-3358.
[20] WANG W, HUANG M, XU X S, et al. Chat more: deepening and widening the chatting topic via a deep model[C]//Proceedings of the 41st International ACM SIGIR Conference, 2018:255-264.
[21] ALEC R, KARTHIK N, TIM S, et al. Improving language understanding by generative pre-training. [EB/OL]. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf[2021-03-15].[22] LU Z, LI H. A Deep architecture for matching short texts[J]. Advances in Neural Information Processing Systems, 2013,5(4): 1367-1375.
[23] DUCHI J, HAZAN E, SINGER Y. Adaptive subgradient methods for online learning and stochastic optimization[J]. The Journal of Machine Learning Research, 2011: 2121-2159.
[24] PAPINENI K, ROUKOS S, WARD T, et al. BLEU: A method for automatic evaluation of machine translation[C]//Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, 2002: 311-318.
[25] LI J, GALLEY M, BROCKETT C, et al. A diversity-promoting objective function for neural conversation models[J]. arXiv:1510.03035,2016.
[26] ZHOU G Y, FANG Y Z, PENG Y H, et al. Neural conversation generation with auxiliary emotional supervised models[J]. ACM Transactions on Asian and Low-resource Language Information Processing, 2020,19(2): 1-17.
[27] XU M H, LI P J, YANG H R, et al. A neural topical expansion framework for unstructured persona-oriented dialogue generation[C]//Proceedings of the ECAI, 2020.
[28] PENG Y, FANG Y, XIE Z, et al. Topic-enhanced emotional conversation generation with attention mechanism[J]. Knowledge-Based Systems, 2019, 163: 429-437.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
国家自然科学基金(61972173)
{{custom_fund}}
PDF(1862 KB)
