语法错误检测是自然语言处理中自动校对技术的重要环节。中文语法灵活多变,而且错别字和语法错误会严重影响其周边范围的词义语义,甚至整个句子的原本含义,另外,现有深度学习模型为提高性能经常引入较多外部信息,也导致训练难度大。因此该研究将语法错误检测视为序列标注任务,提出了一种基于Electra的神经网络模型,以门控双线性神经网络Gated-Bilinear为其下游结构,在预训练语言模型基础上利用相邻Token的特征加强字向量的局部语义相关性,减轻其受到的错误语义影响。研究使用了历年中文语法错误检测(CGED)任务的数据集,训练并评估模型的性能,实验表明,使用该方法的检错性能在单模型和多模型集成方法上均达到最优水平。
Abstract
Grammatical error diagnosis is an important cog in the automatic proofreading technology in natural language processing applications. Chinese grammar rules are flexible, and wrong words and grammatical errors can severely affect the original meanings. This research regarded grammar error diagnosis as a sequence labeling task and proposed a Chinese grammatical error diagnosis model based on Electra and Gated-Bilinear neural network. Based on the pre-trained Transformer language model, Gated-Bilinear neural network takes advantage of the features of nearby tokens to enhance the local semantic correlation and mitigate the impact of grammatical errors. Experiments on NLPTEA Chinese grammatical error diagnosis (CGED) workshop datasets show this model has achieved a state-of-the-art performance on both single model and ensemble learning method.
关键词
中文语法错误检测 /
CGED /
序列标注 /
门控双线性神经网络
{{custom_keyword}} /
Key words
Chinese grammatical error diagnosis /
CGED /
sequence labeling /
gated-bilinear neural network
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] DU?EK O, HOWCROFT D M, RIESER V. Semantic noise matters for neural natural language generation[C]//Proceedings of the 12th International Conference on Natural Language Generation,2019: 421-426.
[2] ZHAO J, LIU H, BAO Z, et al. N-gram model for chinese grammatical error diagnosis[C]//Proceedings of the 4th Workshop on Natural Language Processing Techniques for Educational Applications, 2017: 39-44. .
[3] QIU Z, QU Y. A two-stage model for chinese grammatical error correction[J]. IEEE Access, 2019, 7: 146772-146777.
[4] WANG Y, YUAN R, LUO Y, et al. Chinese grammatical error correction based on hybrid models with data augmentation[C]//Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications. Suzhou, China: Association for Computational Linguistics, 2020: 78-86.
[5] CLARK K, LUONG M-T, LE Q V, et al. Electra: Pre-training text encoders as discriminators rather than generators[J]. arXiv preprint arXiv:2003.10555, 2020.
[6] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conferrence on Neural Information Processing Systems. 2017:6000-6010.
[7] YANG Y, XIE P, TAO J, et al. Alibaba at IJCNLP-2017 Task 1: Embedding grammatical features into lstms for chinese grammatical error diagnosis task[C]//Proceedings of the IJCNLP, Shared Tasks. Taipei, Taiwan: Asian Federation of Natural Language Processing, 2017: 41-46.
[8] FU R, PEI Z, GONG J, et al. Chinese grammatical error diagnosis using statistical and prior knowledge driven features with probabilistic ensemble enhancement[C]//Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications. Melbourne, Australia: Association for Computational Linguistics, 2018: 52-59.
[9] WANG S, WANG B, GONG J, et al. Combining resnet and transformer for chinese grammatical error diagnosis[C]//Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications. Suzhou, China: Association for Computational Linguistics, 2020: 36-43.
[10] LUO Y, BAO Z, LI C, et al. Chinese grammatical error diagnosis with graph convolution network and multi-task learning[C]//Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications. Suzhou, China: Association for Computational Linguistics, 2020: 44-48.
[11] DEVLIN J, CHANG M-W, LEE K, et al. BERT: Pretraining of deep bidirectional transformers for language understanding[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 4171-4186.
[12] CUI Y, CHE W, LIU T, et al. Revisiting pre-trained models for chinese natural language processing[C]//Proceedings of the Association for Computational Linguistics: EMNLP, 2020: 657-668.
[13] BA J L, KIROS J R, HINTON G E. Layer Normalization[J]. arXiv preprint arXiv:1607.06450,2016.
[14] WANG W, BI B, YAN M, et al. Structbert: Incorporating language structures into pre-training for deep language understanding[C]//Proceedings of ICLR, 2020: 1-10.
[15] RAO G, YANG E, ZHANG B. Overview of NLPTEA-2020 shared task for Chinese grammatical error diagnosis[C]//Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications, 2020: 25-35.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
国家自然科学基金(62066007)
{{custom_fund}}
PDF(5074 KB)
