本文提出了一个基于n-gram语言模型进行文本表示,采用链状朴素贝叶斯分类器进行分类的中文文本分类系统。介绍了如何用n-gram语言模型进行文本表示,阐述了链状朴素贝叶斯分类器与n-gram语言模型相结合的优势,分析了n-gram语言模型参数的选取,讨论了分类系统的若干重要问题,研究了训练集的规模和质量对分类系统的影响。根据863计划文本分类测评组所提供的测试标准、训练集以及测试集对本文所设计的分类系统进行测试,实验结果表明该分类系统有良好的分类效果。
Abstract
An automatic Chinese text categorization method based on n-gram language model and chain augmented na?ve Bayesian classifier is proposed. The paper introduces the representation of a text through n-gram language model, argues the advantage of combining n-gram language model and chain augmented na?ve Bayesian classifier, analyzes how to choose the parameters of n-gram language model, and discusses some crucial problems of the categorization system. The effect of quantity and quality of training corpus on classifier performance is also studied experimentally. The categorization system is tested on the 863-project data set for Chinese text categorization. The experimental result shows that the system performs well.
关键词
计算机应用 /
中文信息处理 /
中文文本分类 /
n-gram语言模型 /
链状朴素贝叶斯分类器
{{custom_keyword}} /
Key words
computer application /
Chinese information processing /
Chinese text categorization /
n-gram language model /
chain augmented na?ve Bayesian classifier
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] Ricardo Baeza-Yates, Berthier Ribeiro-Neto, Modern Information Retrieval. [M] China Machine Press, 2003.
[2] Fuchun Peng, Dale Schuurmans, Shaojun Wang, Augmenting Na?ve Bayes Classierswith Statistical Language Models[M]. School of Computer Science at University of Waterloo, 2004.
[3] http://www.863data.org.cn / [OL].
[4] D. Hiemstra, Using Language Models for Information Retrieval [D]. Centre for Telematics and Information Technology, University of Twente, 2001.
[5] A. McCallum, K. Nigam, A Comparison of Event Models for Na?ve Bayes Text Classification[R]. In: proceedings of AAA I-98 Workshop on“Learning for Text Categorization”, 1998.
[6] D. Holmes, R. Forsyth, The Federalist Revisited : New Directions in Authorship Attribution [J]. Literary and linguistic Computing, 1995 (10) : 111 - 127.
[7] J. Ponte, W. Croft, A Language Modeling Approach to Information Retrieval[A]. In : proceeding of ACM Research and Development in Information Retrieval (SIGIR) [C] , 1998.
[8] 刘斌,黄铁军,等. 一种新的基于统计的自动文本分类方法[J]. 中文信息学报, 2002, 16 (6) : 18 - 24.
[9] 刘静,尹存燕,等. 一种规则和贝叶斯方法相结合的文本自动分类策略[J]. 计算机应用研究, 2005, (7) : 88 - 90, 93.
[10] 周水庚,关佶红,等. 基于N-gram信息的中文文档分类研究[J]. 中文信息学报, 2000, 15 (1) : 34 - 39.
[11] 黄友平,史忠植,等. 基于信息几何构建朴素贝叶斯分类器[J]. 通讯和计算机, 2005, (2).
[12] 周颜军,王双成,等. 基于贝叶斯网络的分类器研究[J]. 东北师大学报(自然科学版) , 2003, (2) : 25 - 31.
[13] 黄科,马绍平,等. 基于统计分词的中文网页分类[J]. 中文信息学报, 2002, 16 (6) : 25 - 31.
[14] 周茜,赵明生,等. 中文文本分类中的特征选择研究[J]. 中文信息学报, 2004, 18 (3) : 17 - 23.
[15] http://www.lemurproject.org[OL].
[16] 张王番,多种策略改进朴素贝叶斯分类器[J]. 微机发展, 2005, (4) : 37 - 38, 41.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
国家自然科学基金资助项目(60475007)
{{custom_fund}}
PDF(403 KB)
