基于统计的中文词法分析往往依赖大规模标注语料,语料的规模和质量直接影响词法分析系统的性能。高覆盖率、高质量的语料资源非常有限,而且适用于不同领域的语料往往具有不同的分词和词性标注标准,难以直接混合使用,从而导致既有资源未能充分利用,分词精度下降等问题。针对该问题,该文提出了简单有效的异种语料的自动融合方法,并通过实验验证了提案方法的有效性、较强的实用性以及对多种语料融合的可扩展性。
Abstract
Large scale manually annotated corpora are usually used in research on statistical Chinese lexical analysis. The scale and quality of corpora affect the performance of statistical lexical analysis directly. Corpora in high quality and high rate of coverage are very valuable but limited, and it is very difficult to combine corpora of different domains directly since they are different in segmentation and part of speech (POS) tagging standards. These problems make it difficult to utilize existing resources and prevent the performance improvment in Chinese lexical analysis. To address this issue, this paper presents a simple but effective strategy to optimize the performance and domain adaptability of Chinese lexical analysis by merging different domains corpora automatically. Our experiments verify the validity, the stronger practicability, and the scalability to multiple corpora of the proposed method.
Key wordslexical analysis; merging corpora; domain adaptation
关键词
词法分析 /
语料融合 /
领域适应
{{custom_keyword}} /
Key words
lexical analysis /
merging corpora /
domain adaptation
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] Wenbin Jiang, Liang Huang, Qun Liu. Automatic Adaptation of Annotation Standards: Chinese Word Segmentation and POS Tagging—A Case Study. Association for Computational Linguistics[C]//Proceed-ings of the 47th Annual Meeting of the Association for Computational Linguistics. Suntec, Singapore: ACL Publication Chairs, 2009:522-530.
[2] Hwee Tou Ng, Jin Kiat Low. Chinese part-of-speech tagging: One-at-a-time or all-at-once? word-based or character-based?[C]//Proceedings of Conference on Empirical Methods in Natural Language Processing. Barcelona, Spain: ENMLP Publication Chairs, 2004.
[3] Wenbin Jiang, Liang Huang, Yajuan Lv, et al. A cascaded linear model for joint Chinese word segmentation and part-of-speech tagging[C]//Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics. Oho, USA: ACL Publication Chairs, 2008:897-904.
[4] Wenbin Jiang, Haitao Mi, Qun Liu. Word Lattice Reranking for Chinese Word Segmentation and Part-of-Speech Tagging[C]//Proceedings of the 22nd International Conference on Computational Linguistics. Manchester, England: COLING Publication Chairs, 2008:385-392.
[5] Kun Wang, Chengqing Zong, Keh-Yih Su. A Character-Based Joint Model for Chinese Word Segmentation[C]//Proceedings of the 24th International Conference on Computational Linguistics. Beijing, China: COLING Publication Chairs, 2010:1173-1181.
[6] Zhongguo Li, Maosong Sun. Punctuation as Implicit Annotations for Chinese Word Segmentation[J].Computational Linguistics. Proceedings of Computational Linguistics. 2009, 35(4):505-512.
[7] Yue Zhang, Stephen Clark. Chinese segmentation with a word-based perceptron algorithm[C]//Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics. Prague, Czech Republic: ACL Publication Chairs, 2007:840-847.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
国家自然科学基金资助项目(60873167,60736014);中央高校基本科研业务费专项资金项目(2009JBM027)
{{custom_fund}}
PDF(1742 KB)
