复述生成技术是自然语言处理领域重要的研究方向,具有广泛的应用场景。目前的预训练模型能够很好地从文本中获取丰富的语义信息,但这些模型生成的复述句在语法结构上缺乏多样性。为解决上述问题,该文对比了不同预训练模型在复述生成任务上的效果,选择UniLM预训练模型作为基础模型,并在此基础上提出了新的句子复述生成方法。首先提出了一种构建句法模板的方法,在不改变模板句的句法结构的前提下,使用特殊字符替换模板句中相关词性的词,同时提出编辑向量的方法用于增强预训练模型。实验结果表明,在Quora和ParaNMT-small数据集上,该文提出的模型在自动评价和人工评价指标上均有明显提升。
Abstract
Paraphrase generation technology is an important research direction in natural language processing. Current pre-trained models fail to generate the paraphrases with diverse syntactic structures. This paper proposes a sentence paraphrase generation method based on UniLM pre-trained model. Firstly, we propose a method for constructing syntax templates, which uses special characters to replace relevant part-of-speech words without modifying the syntactic structure in the templates. Also, we propose edit vectors to enhance the pre-trained model. Experiments on the Quora and ParaNMT-small datasets demonstrate the improvements of this method in both automatic and human evaluation metric.
关键词
复述生成 /
预训练模型 /
多样性
{{custom_keyword}} /
Key words
paraphrase generation /
pre-trained model /
diversity
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] CHO K, VAN M B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[J]. arXiv preprint arXiv: 1406.1078, 2014.
[2] FADER A,ZETTLEMOYER L, ETZIONI O. Open question answering over curated and extracted knowledgebases[C]//Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2014: 1156-1165.
[3] SHINYAMA Y, SEKINE S, SUDO K, et al. Automatic paraphrase acquisition from news articles[C]//Proceedings of HLT. San Diego, US, 2002.
[4] CAO R, ZHU S, YANG C, et al. Unsupervised dual paraphrasing for two-stage semantic parsing[J].arXiv preprint arXiv: 2005.13485, 2020.
[5] IYYER M, WIETING J, GIMPEL K, et al. Adversarial example generation with syntactically controlled paraphrase networks[J].arXiv preprint arXiv: 1804.06059, 2018.
[6] KAZEMNEJAD A, SALEHI M, BAGHSHAH M S. Paraphrase generation by learning how to edit from samples[C]//Proceedings of the 58th Annual Meetingof the Association for Computational Linguistics, 2020: 6010-6021.
[7] CHEN M, TANG Q, WISEMAN S, et al. Controllable paraphrase generation with a syntactic exemplar[J].arXiv preprint arXiv: 1906.00565, 2019.
[8] KUMAR A, AHUJA K,VADAPALLI R, et al. Syntax-guided controlled generation of paraphrases[J]. Transactions of the Association for Computational Linguistics, 2020, 8: 330-345.
[9] SUN J, MA X, PENG N. AESOP: Paraphrase generation with adaptive syntactic control[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021: 5176-5189.
[10] YANG H, LAM W, LI P. Contrastive representation learning for exemplar-guided paraphrase generation[J].arXiv preprint arXiv: 2109.01484, 2021.
[11] LIU M, YANG E,XIONG D, et al. Exploring biling-ual parallel corpora for syntactically controllable paraphrase generation[C]//Proceedings of the 29th International Conference on International Joint Conferences on Artificial Intelligence, 2021: 3955-3961.
[12] YANG K, LIU D, LEI W, et al. GCPG: A general framework for controllable paraphrase generation[C]//Proceedings of the Association for Computational Linguistics, 2022: 4035-4047.
[13] DONG L, YANG N, WANG W H, et al. Unified language model pre-training for natural language understanding and generation[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019: 13063-13075.
[14] PAPINENI K, ROUKOS S, WARD T, et al. BLEU: A method for automatic evaluation of machine translation[C]//Proceedings of the 40th Annual Meeting ofthe Association for Computational Linguistics, 2002: 311-318.
[15] SNOVER M, DORR B, SCHWARTZ R, et al. A study of translation edit rate with targeted human annotation[C]//Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical papers, 2006: 223-231.
[16] BUI T C, LE V D, TO H T, et al. Generative pretraining for paraphrase generation by representing and predicting spans in exemplars[C]//Proceedings of the IEEE International Conference on Big Data and Smart Computing, 2021: 83-90.
[17] PENNINGTON J,SOCHER R, MANNING C D. Glove: Global vectors for word representation[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2014: 1532-1543.
[18] GUU K, HASHIMOTO T B, OREN Y, et al. Generating sentences by editing prototypes[J]. Transactions of the Association for Computational Linguistics, 2018, 6: 437-450.
[19] WIETING J, GIMPEL K. Pushing the limits of paraphrastic sentence embeddings with millions of mac-hine translations[J]. arXiv preprint arXiv: 1711.05732, 2017.
[20] KINGMA D P, BA J. Adam: A method for stochastic optimization[J]. arXiv preprint arXiv: 1412.6980, 2014.
[21] BANERJEE S,LAVIE A. An automatic metric for mt evaluation with improved correlation with human judgments[C]//Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization, 2005: 65-72.
[22] LIN C Y. Rouge: A package for automatic evaluation of summaries[C]//Proceedings of the Text Summarization Branches Out, 2004: 74-81.
[23] ZHANG K,SHASHA D. Simple fast algorithms for the editing distance between trees and related problems[J]. SIAM Journal on Computing, 1989, 18(6): 1245-1262.
[24] LEWIS M, LIU Y,GOYAL N, et al. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[J]. arXiv preprint arXiv: 1910.13461, 2019.
[25] VASWANI A. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017: 5998-6008.
[26] DEVLIN J, CHANG M W, LEE K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J].arXiv preprint arXiv: 1810.04805, 2018.
[27] LUND, BRADY D, Wang T. “Chatting about ChatGPT: How may AI and GPT impact academia and libraries?”[J] Library hi Tech News, 2023: 26-29.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
国家自然科学基金(61702234);船舶总体性能创新研究开放基金(25422217)
{{custom_fund}}
PDF(2793 KB)
