语法纠错任务是自然语言处理领域的一项重要任务,近年来受到了学术界广泛关注。该任务旨在自动识别并纠正文本中所包含的语法、拼写以及语序错误等。该文将语法纠错任务看作是翻译任务,即将带有错误表达的文本翻译成正确的文本,采用基于多头注意力机制的Transformer模型作为纠错模型,并提出了一种动态残差结构,动态结合不同神经模块的输出来增强模型捕获语义信息的能力。受限于目前训练语料不足的情况,该文提出了一种数据增强方法,通过对单语语料的腐化从而生成更多的纠错数据,进一步提高模型的性能。实验结果表明,该文所提出的基于动态残差的模型增强以及腐化语料的数据增强方法对纠错性能有着较大的提升,在NLPCC 2018中文语法纠错共享评测数据上达到了最优性能。
Abstract
Grammatical error correction is an important task in the field of natural language processing, which has attracted wide attention in recent years. This paper regards grammatical error correction task as a translation task to translate the wrong texts into the right ones. We use the transformer model with multi-head attention mechanism as framework, and propose a dynamic residual structure to combine the outputs of different neural blocks dynamically to better capture semantic information. Due to the lack of training corpus, we propose a data augmentation method to generate the parallel data by corrupting a monolingual corpus. The experimental results show that the proposed method based on dynamic residuals and data augmentation has significantly improved the performance of error correction, achieving the best performance on NLPCC 2018 Chinese grammatical error correction task.
关键词
语法纠错 /
多头注意力 /
动态残差结构 /
数据增强
{{custom_keyword}} /
Key words
grammatical error correction /
multi-head attention /
dynamic residual structure /
data augmentation
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proceedings of Advances in Neural Information Processing Systems, 2017: 5998-6008.
[2] Bustamante F R, León F S.GramCheck: A grammar and style checker[C]//Proceedings of the 16th Conference on Computational Linguistics-Volume 1. Association for Computational Linguistics, 1996: 175-181.
[3] Heidorn G E, Jensen K, Miller L A, et al. The EPISTLE text-critiquing system[J]. IBM Systems Journal, 1982, 21(3): 305-326.
[4] Brockett C, Dolan W B,Gamon M. Correcting ESL errors using phrasal SMT techniques[C]//Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2006: 249-256.
[5] Ng H T, Wu S M, Wu Y, et al. The CoNLL-2013 shared task on grammatical error correction[C]//Proceedings of the 17th Conference on Computational Natural Language Learning: Shared Task, 2013: 1-12.
[6] Ng H T, Wu S M, Briscoe T, et al. The CoNLL-2014 shared task on grammatical error correction[C]//Proceedings of the 18th Conference on Computational Natural Language Learning: Shared Task, 2014: 1-14.
[7] 谭咏梅, 王晓辉, 杨一枭. 基于语料库的英语文章语法错误检查及纠正方法[J]. 北京邮电大学学报, 2016, 39(4):92-97.
[8] Junczys-Dowmunt M, Grundkiewicz R. The AMU system in the CoNLL-2014 shared task: Grammatical error correction by data-intensive and feature-rich statistical machine translation[C]//Proceedings of the 18th Conference on Computational Natural Language Learning: Shared Task, 2014: 25-33.
[9] Koehn P, Hoang H, BirchA, et al. Moses: Open source toolkit for statistical machine translation[C]//Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, 2007: 177-180.
[10] Mizumoto T, Komachi M, Nagata M, et al. Mining revision log of language learning SNS for automated Japanese error correction of second language learners[C]//Proceedings of 5th International Joint Conference on Natural Language Processing, 2011: 147-155.
[11] Felice M, Yuan Z, Andersen E, et al. Grammatical error correction using hybrid systems and type filtering[C]//Proceedings of the 18th Conference on Computational Natural Language Learning: Shared Task, 2014: 15-24.
[12] 谭咏梅, 杨一枭, 杨林,等. 基于LSTM和N-gram的ESL文章的语法错误自动纠正方法[J]. 中文信息学报, 2018, 32(06):24-32.
[13] Chollampatt S, Ng H T. A multilayer convolutional encoder-decoder neural network for grammatical error correction[C]//Proceedings of 32nd AAAI Conference on Artificial Intelligence, 2018.
[14] Grundkiewicz R, Junczys-Dowmunt M. Near human-level performance in grammatical error correction with hybrid machine translation[J]. arXiv:1804.05945, 2018.
[15] 王洁. 计算机识别汉语语法偏误的可行性分析[J]. 语言文字应用, 2011(1):135-142.
[16] 龚小谨, 罗振声, 骆卫华. 中文文本自动校对中的语法错误检查[J]. 计算机工程与应用, 2003, 39(8):98-100.
[17] Gaoqi R, Zhang B, Endong X, et al. IJCNLP-2017 task 1: Chinese grammatical error diagnosis[C]//Proceedings of the IJCNLP 2017, Shared Tasks. 2017: 1-8.
[18] Zhao Y, Jiang N, Sun W, et al. Overview of the NLPCC 2018 shared task: Grammatical error correction[C]//Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing. Springer, Cham, 2018: 439-445.
[19] Fu K, Huang J,Duan Y. Youdao’s winning solution to the NLPCC-2018 Task 2 challenge: A neural machine translation approach to Chinese grammatical error correction[C]//Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing. Springer, Cham, 2018: 341-350.
[20] Zhou J, Li C, Liu H, et al. Chinese grammatical error correction using statistical and neural models[C]//Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing. Springer, Cham, 2018: 117-128.
[21] Ren H, Yang L,Xun E. A Sequence to sequence learning for Chinese grammatical error correction[C]//Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing. Springer, Cham, 2018: 401-410.
[22] Sennrich R, Haddow B, Birch A. Neural machine translation of rare words with subword units[J]. arXiv preprint arXiv:1508.07909, 2015.
[23] Peters M, Neumann M,Iyyer M, et al. Deep contextualized word representations[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018: 2227-2237.
[24] 张宝林. “HSK动态作文语料库”的标注问题[C]. 中文电化教学国际研讨会, 2006.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
北京语言大学语言资源高精尖创新中心项目(TYZ19005);国家语委信息化项目(ZDI135-105,YB135-89)
{{custom_fund}}
PDF(5647 KB)
