该文给出了一种与传统分词不同的词法分析选择,提出了一种利用扩展标记集来实现词内部结构分析的方法。首先阐述了词的内部结构特点,把结构中的前后缀视为特殊的词,进而通过识别出每一个词的前后缀来识别词的内部结构。方法是把词内部结构识别问题转换成序列标注问题,通过扩展标记集,采用CRF模型来实现词的内部结构分析。最终实验表明,无论是在总体性能上,还是在各层结构的识别上都取得了较高的准确度。
Abstract
This paper proposes a different way of lexical analysis, to analyze the internal structures of words, and presents a word structure analysis method by extending the word tag set. First, we describe the characteristics of the internal structures of words, By treating the prefixes and suffixes within words structures as special words, we identify the internal structures of words through the detection of prefixes and suffixes. We convert the issue of identifying the internal structures of words into the sequence tagging problem, adopting the CRF model to realize the words structures analysis using extending the word tag set. The experiment shows that they achieve higher accuracy both on overall performance and on the identification of each layer structure.
关键词
扩展标记集 /
词结构分析 /
前后缀 /
序列标注问题
{{custom_keyword}} /
Key words
extend the word tag set /
words structures analysis /
prefixes and suffixes /
sequence tagging problem
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] Wu A D. Customizable segmentation of morphologically derived words in Chinese[J]. International Journal of Computational Linguistics and Chinese Language Processing, 2003, 8(1): 1-27.
[2] Zhao H. Character-level dependencies in Chinese: Usefulness and learning[C]//Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 2009: 879-887.
[3] Zhang M S, Zhang Y, Che W, et al. Chinese parsing exploiting characters[C]//Proceedings of 51st Annual Meeting of the Association for Computational Linguistics. 2013.
[4] Li Z G. Parsing the internal structure of words: a new paradigm for Chinese word segmentation[C]//Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1. Association for Computational Linguistics, 2011: 1405-1414.
[5] 方艳, 孙静, 丁彬,等. 基于层叠CRF模型的词结构分析[J]. 中文信息学报,已录用.
[6] Li S, Huang C. Word Boundary Decision with CRF for Chinese Word Segmentation[C]//Proceedings of PACLIC-2009. 2009:726-732.
[7] Peng F, Feng F, McCallum A. Chinese segmentation and new word detection using conditional random fields[C]//Proceedings of COLING. 2004.
[8] Xue N W, Converse S P. Combining classifiers for Chinese word segmentation[C]//Proceedings of the first SIGHAN workshop on Chinese language processing-Volume 18. Association for Computational Linguistics, 2002: 1-7.
[9] Zhao H, Huang C N, Li M. An improved Chinese word segmentation system with conditional random field[C]//Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing. Sydney: July, 2006: 108-117.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
国家自然科学基金青年项目(61202162);教育部博士点基金新教师类课题(20123201120011);国家863计划前沿技术研究类项目(2012AA011102)
{{custom_fund}}
PDF(3244 KB)
