王辰成,杨麟儿,王莹莹,杜永萍,杨尔弘. 基于Transformer增强架构的中文语法纠错方法[J]. 中文信息学报, 2020, 34(6): 106-114.
WANG Chencheng, YANG Liner, WANG Yingying, DU Yongping, YANG Erhong. Chinese Grammatical Error Correction Method Based on Transformer Enhanced Architecture. , 2020, 34(6): 106-114.
Chinese Grammatical Error Correction Method Based on Transformer Enhanced Architecture
WANG Chencheng1,2, YANG Liner2,3, WANG Yingying2,3, DU Yongping1, YANG Erhong2,3
1.Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China; 2.Beijing Advanced Innovation Center for Language Resources, Beijing Language and Culture University, Beijing 100083, China; 3.School of Information Science, Beijing Language and Culture University, Beijing 100083, China
Abstract:Grammatical error correction is an important task in the field of natural language processing, which has attracted wide attention in recent years. This paper regards grammatical error correction task as a translation task to translate the wrong texts into the right ones. We use the transformer model with multi-head attention mechanism as framework, and propose a dynamic residual structure to combine the outputs of different neural blocks dynamically to better capture semantic information. Due to the lack of training corpus, we propose a data augmentation method to generate the parallel data by corrupting a monolingual corpus. The experimental results show that the proposed method based on dynamic residuals and data augmentation has significantly improved the performance of error correction, achieving the best performance on NLPCC 2018 Chinese grammatical error correction task.
[1] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proceedings of Advances in Neural Information Processing Systems, 2017: 5998-6008. [2] Bustamante F R, León F S.GramCheck: A grammar and style checker[C]//Proceedings of the 16th Conference on Computational Linguistics-Volume 1. Association for Computational Linguistics, 1996: 175-181. [3] Heidorn G E, Jensen K, Miller L A, et al. The EPISTLE text-critiquing system[J]. IBM Systems Journal, 1982, 21(3): 305-326. [4] Brockett C, Dolan W B,Gamon M. Correcting ESL errors using phrasal SMT techniques[C]//Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2006: 249-256. [5] Ng H T, Wu S M, Wu Y, et al. The CoNLL-2013 shared task on grammatical error correction[C]//Proceedings of the 17th Conference on Computational Natural Language Learning: Shared Task, 2013: 1-12. [6] Ng H T, Wu S M, Briscoe T, et al. The CoNLL-2014 shared task on grammatical error correction[C]//Proceedings of the 18th Conference on Computational Natural Language Learning: Shared Task, 2014: 1-14. [7] 谭咏梅, 王晓辉, 杨一枭. 基于语料库的英语文章语法错误检查及纠正方法[J]. 北京邮电大学学报, 2016, 39(4):92-97. [8] Junczys-Dowmunt M, Grundkiewicz R. The AMU system in the CoNLL-2014 shared task: Grammatical error correction by data-intensive and feature-rich statistical machine translation[C]//Proceedings of the 18th Conference on Computational Natural Language Learning: Shared Task, 2014: 25-33. [9] Koehn P, Hoang H, BirchA, et al. Moses: Open source toolkit for statistical machine translation[C]//Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, 2007: 177-180. [10] Mizumoto T, Komachi M, Nagata M, et al. Mining revision log of language learning SNS for automated Japanese error correction of second language learners[C]//Proceedings of 5th International Joint Conference on Natural Language Processing, 2011: 147-155. [11] Felice M, Yuan Z, Andersen E, et al. Grammatical error correction using hybrid systems and type filtering[C]//Proceedings of the 18th Conference on Computational Natural Language Learning: Shared Task, 2014: 15-24. [12] 谭咏梅, 杨一枭, 杨林,等. 基于LSTM和N-gram的ESL文章的语法错误自动纠正方法[J]. 中文信息学报, 2018, 32(06):24-32. [13] Chollampatt S, Ng H T. A multilayer convolutional encoder-decoder neural network for grammatical error correction[C]//Proceedings of 32nd AAAI Conference on Artificial Intelligence, 2018. [14] Grundkiewicz R, Junczys-Dowmunt M. Near human-level performance in grammatical error correction with hybrid machine translation[J]. arXiv:1804.05945, 2018. [15] 王洁. 计算机识别汉语语法偏误的可行性分析[J]. 语言文字应用, 2011(1):135-142. [16] 龚小谨, 罗振声, 骆卫华. 中文文本自动校对中的语法错误检查[J]. 计算机工程与应用, 2003, 39(8):98-100. [17] Gaoqi R, Zhang B, Endong X, et al. IJCNLP-2017 task 1: Chinese grammatical error diagnosis[C]//Proceedings of the IJCNLP 2017, Shared Tasks. 2017: 1-8. [18] Zhao Y, Jiang N, Sun W, et al. Overview of the NLPCC 2018 shared task: Grammatical error correction[C]//Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing. Springer, Cham, 2018: 439-445. [19] Fu K, Huang J,Duan Y. Youdao’s winning solution to the NLPCC-2018 Task 2 challenge: A neural machine translation approach to Chinese grammatical error correction[C]//Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing. Springer, Cham, 2018: 341-350. [20] Zhou J, Li C, Liu H, et al. Chinese grammatical error correction using statistical and neural models[C]//Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing. Springer, Cham, 2018: 117-128. [21] Ren H, Yang L,Xun E. A Sequence to sequence learning for Chinese grammatical error correction[C]//Proceedings of the CCF International Conference on Natural Language Processing and Chinese Computing. Springer, Cham, 2018: 401-410. [22] Sennrich R, Haddow B, Birch A. Neural machine translation of rare words with subword units[J]. arXiv preprint arXiv:1508.07909, 2015. [23] Peters M, Neumann M,Iyyer M, et al. Deep contextualized word representations[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018: 2227-2237. [24] 张宝林. “HSK动态作文语料库”的标注问题[C]. 中文电化教学国际研讨会, 2006.