词汇约束文本生成是自然语言处理领域的重要研究任务之一,旨在给定一组有序词汇,生成包含这些词汇的流畅文本,在语言教学、文本生成、信息检索等领域有广泛应用。现有的生成方法存在生成速度慢、无法包含所有约束词等问题,难以满足实际应用需求。该文提出一种基于片段预测的端到端词汇受限文本生成方法,将词汇约束文本生成视为对约束词之间的文本片段的预测,利用基于二维位置编码的预训练语言模型预测所有片段,再将其填充回约束词的对应位置,从而保证了生成速度和词汇约束;利用词性标注方式构造多参考数据进行数据增强,进一步提升了文本生成质量。为验证方法的有效性,该文在公开的英文数据集,以及基于国际中文教材构建的中文数据集上进行了实验,结果表明,该文提出的LCTG-SP方法可以满足所有词汇约束、具有较快生成速度,生成文本的流利度和多样性表现更好。本文中的模型代码和数据开源在GitHub上①。
Abstract
Lexically constrained text generation aims to generate fluent text containing these words given a set of ordered words, which is widely used in language teaching, text generation, information retrieval, and other fields. This paper proposes an end-to-end lexically constrained text generation method based on fragment prediction, which considers the lexically constrained text generation task as an end-to-end prediction of text fragments between constrained words. It uses two-dimensional position encoding to learn semantic relationships between segments and within segments, thereby speeding up text generation while ensuring generation quality and lexical constraints. In addition, the part-of-speech tagging method is used to construct multi-reference data for data augmentation. Experiments are conducted on the English dataset publicly available and a Chinese dataset of international Chinese textbooks constructed by this paper. The experimental results show that the method proposed in this paper has significantly improved generation speed, fluency, and diversity(code and data available at https://github.com/blcuicall/LCTG-SP).
关键词
词汇约束 /
片段预测 /
文本生成 /
数据增强
{{custom_keyword}} /
Key words
lexical constraints /
segment prediction /
text generation /
data augmentation
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] HU Z, YANG Z, LIANG X, et al. Toward controlled generation of text[C]//Proceedings of the International Conference on Machine Learning, 2017: 1587-1596.
[2] TANG H, LI M, JIN B. A topic augmented text generation model: joint learning of semantics and structural features[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019: 5090-5099.
[3] HE X, LI V O K. Show me how to revise: Improving lexically constrained sentence generation with xlnet[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(14): 12989-12997.
[4] CHEN H, YI X, SUN M, et al. Sentiment-controllable Chinese poetry generation[C]//Proceedings of the International Joint Conference on Artificial Intelligence, 2019: 4925-4931.
[5] SHAO Y, SHAO T, WANG M, et al. A sentiment and style controllable approach for Chinese poetry generation[C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management, 2021: 4784-4788.
[6] ZHAO C, WALKER M, Chaturvedi S. Bridging the structural gap between encoding and decoding for data-to-text generation[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 2481-2491.
[7] WANG Y, WOOD I, WAN S, et al. Mention flags (MF): Constraining transformer-based text generators[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 103-113.
[8] MIAO N, ZHOU H, MOU L, et al. Cgmh: Constrained sentence generation by metropolis-hastings sampling[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(01): 6834-6842.
[9] HOKAMP C, LIU Q. Lexically constrained decoding for sequence generation using grid beam search[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017: 1535-1546.
[10] LIU X, PAN S, ZHANG Q, et al. Generating keyword queries for natural language queries to alleviate lexical chasm problem[C]//Proceedings of the 27th ACM International Conference on Information and Knowledge Management, 2018: 1163-1172.
[11] YU S, LIU J, YANG J, et al. Few-shot generative conversational query rewriting[C]//Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020: 1933-1936.
[12] LIU X, HU J, SHEN Q, et al. Geo-BERT pre-training model for query rewriting in POI search[C]//Proceedings of the Association for Computational Linguistics: EMNLP, 2021: 2209-2214.
[13] YAO L, PENG N, WEISCHEDEL R, et al. Plan-and-write: Towards better automatic storytelling[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(01): 7378-7385.
[14] CARO K, MENDINUETA N R. Lexis, lexical competence and lexical knowledge: A review[J]. Journal of Language Teaching & Research, 2017, 8(2): 125-138.
[15] POST M, VILAR D. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation[C]//Proceedings of NAACL, 2018: 1314-1324.
[16] SHA L. Gradient-guided unsupervised lexically constrained text generation[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2020: 8692-8703.
[17] BERGLUND M, RAIKO T, HONKALA M, et al. Bidirectional recurrent neural networks as generative models[J]//Proceedings of the 28th International Conference on Neural Information Processing Systems, 2015: 856-864.
[18] WANG A, CHO K. BERT has a mouth, and it must speak: BERT as a Markov random field language model[J]//Proceedings of NAACL, 2019: 30-36.
[19] DEVLIN J, CHANG M W, LEE K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of NAACL, 2019: 4171-4186.
[20] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6000-6010.
[21] ZHANG Y, WANG G, LI C, et al. POINTER: Constrained progressive text generation via insertion-based generative pre-training[C]//Proceedings of EMNLP, 2020: 8649-8670.
[22] DU Z, QIAN Y, LIU X, et al. GLM: General language model pretraining with autoregressive blank infilling[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022: 320-335.
[23] QIN L, WELLECK S, KHASHABI D, et al. Cold decoding: Energy-based constrained text generation with langevin dynamics[C]//Proceedings of NeurLPS, 2022: 1-14.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
PDF(1888 KB)
