1.MOE Key Laboratory of Computational Linguistics, Peking University, Beijing 100871, China; 2.Department of Chinese Language and Literature, Peking University, Beijing 100871, China; 3.Center for Chinese Linguistics, Peking University, Beijing 100871, China
Abstract：This paper proposes a fine-grained evaluation scheme on Chinese POS Tagging. The key to this task is to determine the evaluation items and the samples (words) for each item. This paper presents an evaluation set of 5 873 sentences, totaling 2 326 words for 70 evaluation items. Several common open source POS taggers are evaluated. Finally, this paper discusses the advantages of the merits of this evaluation approach, especially in contrast to the classical methods.
[1] Lehmann S,Oepen S, Regnier-Prost S, et al. Tsnlp: Test suites for natural language processing[C]//Proceedings of the 16th Conference on Computational Linguistics-Volume 2. Association for Computational Linguistics, 1996: 711-716. [2] King M,Falkedal K. Using test suites in evaluation of machine translation systems[G]. COLNG 1990 Volume 2: Papers Presented to the 13th International Conference on Computational Linguistics,1990: 211-216. [3] Cooper R, Crouch D, VanEijck J, et al. Using the framework[R]. Technical Report LRE 62-051 D-16, The FraCaS Consortium, 1996. [4] Elkahky A, Webster K, Andor D, et al. A challenge set and methods for moun-verb ambiguity[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018: 2562-2572. [5] Belinkov Y, Glass J. Analysis methods in neural language processing: A survey[J]. Transactions of the Association for Computational Linguistics, 2019, 7: 49-72. [6] 刘金宁. 词性标注体系对中文分词的影响[D].大连: 大连理工大学硕士学位论文,2010. [7] 杨尔弘,方莹,刘冬明,等.汉语自动分词和词性标注评测[J].中文信息学报,2006,20(01): 44-49,97. [8] 俞士汶,段慧明,朱学锋,等. 北大语料库加工规范: 切分· 词性标注· 注音[J]. 汉语语言与计算学报,2003,13(2): 121-158. [9] Luo R, Xu J, Zhang Y, et al. PKUSEG: A toolkit for multi-domain Chinese word segmentation[J].arXiv preprint arXiv: 1906.11455, 2019. [10] Sun M, Chen X, Zhang K, et al.Thulac: An efficient lexical analyzer for Chinese[CP/OL]. 2016-01-10. https://git.hub.com/thanlp/THUL AL. [11] Brants T. TnT: a statistical part-of-speech tagger[C]//Proceedings of the 6th Conference on Applied Natural Language Processing. Association for Computational Linguistics, 2000: 224-231. [12] Wang K,Zong C, Su K Y. Which is more suitable for Chinese word segmentation, the generative model or the discriminative one?[C]//Proceedings of the 23rd Pacific Asia Conference on Language, Information and Computation, Volume 2,2009: 827-834. [13] Lample G, Ballesteros M, Subramanian S, et al. Neural architectures for named entity recognition[J]. arXiv preprint arXiv: 1603.01360, 2016. [14] Zhang H P, Yu H K,Xiong D Y, et al. HHMM-based Chinese lexical analyzer ICTCLAS[C]//Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing-Volume 17. Association for Computational Linguistics, 2003: 184-187. [15] 宋柔.关于分词规范的探讨[J].语言文字应用,1997(03): 113-114.
2020, Vol. 34 
