中文自动分词是中文信息检索中预处理工作的一部分,也是中文信息检索技术中的重要问题之一。针对在信息检索中完整词整体表达更有意义、更能体现用户查询目的的问题,结合完整词的成词特点,将互信息和完整词前后缀的计算,与组成完整词的可信度相关联,提出基于可信度的三种中文完整词自动识别方法,分别构成基于全信度、偏信度,以及前两者加权平均的混信度的完整词识别方法,设计及实现了基于可信度的三种完整词自动识别中文分词原型系统。最后给出了对第二届SIGHAN(2005)北京大学测试集语料的各项实验测试结果和分析,结果表明该原型系统的识别性能良好,且能同时满足多种性能的需求。

To identify the word is an elementary preprocessing for Chinese information retrieval. To capture the semantic integrity and the intension of the user query, this paper analyzes the characteristics of Chinese integrated word and presents three methods for its automatic detection by combining the mutual information, the prefix-and-suffix information of integrity word and the confidence for the integrated word. We further design and realize three prototype systems for Chinese integrated word identification based on the proposed overall-confidence, partial-confidence and weighted joint-confidence methods respectively. Finally, experiments on the 2^nd SIGHAN(2005) PKU test corpus show that the performance of the system is good, capable of satisfying demands on different aspects of performance.
Key words computer application; Chinese information processing; Chinese word segmentation; mutual information; reliability; automatic recognition