基于复述模型的词语替代方法

PDF(2863 KB)

中文信息学报 ›› 2023, Vol. 37 ›› Issue (5) : 22-31,43.

语言分析与计算

基于复述模型的词语替代方法

强继朋¹,陈宇¹,李杨¹,李云¹,吴信东^2,3

作者信息 +

Lexical Substitution Based on Paraphrase Modeling

QIANG Jipeng¹, CHEN Yu¹, LI Yang¹, LI Yun¹, WU Xindong^2,3

Author information +

History +

摘要

词语替代任务旨在为句子中的目标词寻找合适的替代词。基于预训练语言模型BERT的词语替代方法直接利用目标词的上下文信息生成替代候选词。由于标注数据资源的缺乏使得研究人员通常采用无监督的方法,这也限制了预训练模型在此任务上的适用性。考虑到现有的大规模复述语料中包含了大量的词语替代规则,该文提出一种通过复述模型生成替代候选词的方法。具体的做法是: 利用复述语料训练一个神经复述模型;提出了一种只关注目标词变化的解码策略,用于从复述模型中生成替代词;根据文本生成评估指标计算替代词对原句意思的改变程度,对替代词排序。相对已有的词语替代方法,在两个广泛使用的数据集LS07和CoInCo上进行评估,该文提出的方法取得了显著的提高。

Abstract

Lexical substitution (LS) aims at finding an appropriate substitute for a target word in a sentence. In contrast to the BERT-based LS, this paper proposes a method to generate substitution candidates base on paraphrase to utilize the existing large-scale paraphrase corpus which contains a large number of rules of word substitution. Specifically, we first employ a paraphrase dataset to train a neural paraphrase model. Then, we propose a special decoding method to focus only on the variation of the target word to extract substitute candidates. Finally, we rank substitute candidates for choosing the most appropriate substitution without modifying the meaning of the original sentence based on text generation evaluation metrics. Compared with existing state-of-the-art methods, experimental results show that our proposed methods achieve the best results on two widely used benchmarks (LS07 and CoInCo).

导出引用

强继朋, 陈宇, 李杨, 李云, 吴信东. 基于复述模型的词语替代方法. 中文信息学报. 2023, 37(5): 22-31,43

QIANG Jipeng, CHEN Yu, LI Yang, LI Yun, WU Xindong. Lexical Substitution Based on Paraphrase Modeling. Journal of Chinese Information Processing. 2023, 37(5): 22-31,43

参考文献

[1] HINTZ G, BIREMANN C. Language transfer learning for supervised lexical substitution[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016: 118-129.
[2] ZHOU W, GE T, XU K, et al. BERT-based lexical substitution[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019: 3368-3373.
[3] AREFYEV N, SHELUDKO B, PODOLSKIY A, et al. A comparative study of lexical substitution approaches based on neural language models[J]. arXiv preprint arXiv:2006.00031, 2020.
[4] PEATZOLD G, SPECIA L. Unsupervised lexical simplification for non-native speakers[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2016: 3761-3767.
[5] QIANG J, LI Y, ZHU Y, et al. Lexical simplification with pretrained encoders[C]//Proceedings of the AAAI, 2020: 8649-8656.
[6] 强继朋, 钱镇宇, 李云,等. 基于预训练表示模型的英语词语简化方法[J]. 自动化学报, 2022, 48(5):1001-1013.
[7] 强继朋, 李云, 吴信东. 自动词语简化方法综述[J]. 中文信息学报, 2021, 35(12):1-16.
[8] MCCARTHY D. Lexical substitution as a task for wsd evaluation[C]//Proceedings of the ACL Workshop on Word Sense Disambiguation: Recent Successes and Future Directions, 2002: 89-115.
[9] LEE M, DONAHUE C, JIA R, et al. Swords: A benchmark for lexical substitution with improved data coverage and quality[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021: 4362-4379.
[10] HASSAN S, CSOMAI A, BANER C, et al. Unt: Subfinder: Combining knowledge sources for automatic lexical substitution[C]//Proceedings of the 4th International Workshop on Semantic Evaluations, 2007: 410-413.
[11] YURET D. KU: Word sense disambiguation by substitution[C]//Proceedings of the 4th International Workshop on Semantic Evaluations, 2007: 207-214.
[12] MELAMUD O, DAGAN I, GOLDBERGER J. Modeling word meaning in context with substitute vectors[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2015: 472-482.
[13] MICHALOPOULOS G, MCKILLOP I, WONG A, et al. LexSubCon: Integrating knowledge from lexical resources into contextual embeddings for lexical substitution[J]. arXiv preprint arXiv:2107.05132, 2021.
[14] LACERRA C, TRIPODI R, NAVIGLI R. GENESIS: A generative approach to substitutes in context[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021: 10810-10823.
[15] PAVLICK E, CALLISON-BURCH C. Simple PPDB: A paraphrase database for simplification[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016: 143-148.
[16] KRIZ R, MILTSAKAKI E, APIDIANAKI M, et al. Simplification using paraphrases and context-based lexical substitution[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018: 207-217.
[17] GANITKEVITCH J, VAN DURME B, CALLISON-BURCH C. PPDB: The paraphrase database[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2013: 758-764.
[18] PAVLICK E, RASTOGI P, GANITKEVITCH J, et al. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 2015: 425-430.
[19] LEWIS M, LIU Y, GOYAL N, et al. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[J]. arXiv preprint arXiv:1910.13461, 2019.
[20] HU J E, SINGH A, HOLZENBERGER N, et al. Large-scale, diverse, paraphrastic bitexts via sampling and clustering[C]//Proceedings of the 23rd Conference on Computational Natural Language Learning, 2019: 44-54.
[21] ZHANG T,KISHORE V,Wu F,et al. BERTScore: Evaluating text generation with BERT[C]//Proceedings of the International Conference on Learning Representations, 2019.
[22] SELLAM T, DAS D, PARIKH A P. BLEURT: Learning robust metrics for text generation[J]. arXiv preprint arXiv:2004.04696, 2020.
[23] SZARVAS G, BUSA-FEKETE R, HULLERMEIER E. Learning to rank lexical substitutions[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2013: 1926-1932.
[24] MELAMUD O, LEVY O, DAGAN I. A simple word embedding model for lexical substitution[C]//Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, 2015: 1-7.
[25] CHO K. Noisy parallel approximate decoding for conditional recurrent language model[J]. arXiv preprint arXiv:1605.03835, 2016.
[26] LI D, ZHANG Y, PENG H, et al. Contextualized perturbation for textual adversarial attack[J]. arXiv preprint arXiv:2009.07502, 2020.
[27] TAM Y C. Cluster-based beam search for pointer-generator chatbot grounded by knowledge[J]. Computer Speech & Language, 2020, 64: 101094.
[28] VIJAYAKUMAR A K, COGSWELL M, SELVARAJU R R, et al. Diverse beam search for improved description of complex scenes[C]//Proceedings of the AAAI, 2018.
[29] MENG Y, AO X, He Q, et al. ConRPG: Paraphrase generation using contexts as regularizer[J]. arXiv preprint arXiv:2109.00363, 2021.
[30] KADOTANI S, KAJIWARE T, ARASE Y, et al. Edit distance based curriculum learning for paraphrase generation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, 2021: 229-234.
[31] ZHANG Y, VOGEL S, WAIBEL A. Interpreting BLEU/NIST scores: How much improvement do we need to have a better system?[C]//Proceedings of the 4th International Conference on Language Resources and Evaluation, 2004.[32] AGIRRE E, MARQUEZ L, WCENTOWSKI R. Proceedings of the Fourth International Workshop on Semantic Evaluations[C]//Proceedings of the 4th International Workshop on Semantic Evaluations,2007.
[33] KREMER G, ERK K, PADO S, et al. What substitutes tell us-analysis of an “all-words” lexical substitution corpus[C]//Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. 2014: 540-549.
[34] LIN Y, AN Z, WU P, et al. Improving contextual representation with gloss regularized pre-training[C]//Proceedings of the Association for Computational Linguistics. 2022: 907-920.
[35] SZARVAS G, BUSA-FEKETE R, HULLERMEIER E. Learning to rank lexical substitutions[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing. 2013: 1926-1932.
[36] AREFYEV N, SHELUDKO B, PODOLSKIY A, et al. A comparative study of lexical substitution approaches based on neural language models[J]. arXiv preprint arXiv:2006.00031, 2020.
[37] MELAMUD O, GOLDBERGER J, DAGAN I. Context2vec: Learning generic context embedding with bidirectional lstm[C]//Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning. 2016: 51-61.
[38] QIANG J, LI Y, ZHU Y, Yunhao Yuan, et al. LSBert: Lexical Simplification Based on BERT. IEEE/ACM Transactions on Audio, Speech and Language Processing, 2021, (99): 1-7.
[39] QIANG J, LU X, LI Y, et al. Chinese Lexical Simplification[J]. 10.48550/arXiv.2010.07048,2020.

基金

国家自然科学基金(62076217,61703362);扬州大学“青蓝工程”资助项目

PDF(2863 KB)

Accesses

Citation

Detail

段落导航

摘要
Abstract
关键词
Key words
引用本文
参考文献
基金

选择文件类型/文献管理软件名称

选择包含的内容

摘要

Abstract

关键词

Key words

引用本文

{{custom_sec.title}}

{{custom_sec.title}}

参考文献

{{custom_fnGroup.title_cn}}

脚注

基金

Published
2023-07-08
Issue Date
2023-07-08