本文介绍了一种汉语自动分词软件系统,该系统对原文进行三遍扫描:第一遍,利用切分标记将文本切分成汉字短串的序列;第二遍,根据各短串的每个子串在上下文中的频度计算其权值,权值大的子串视为候选词;第三遍,利用候选词集和一部常用词词典对汉字短串进行切分。实验表明,该分词系统的分词精度在1.5%左右,能够识别大部分生词,特别适用于文献检索等领域。

This paper present s a software system on Chinese automatic word segmentation. The original text is scanned three times :first ,the text is cut into short Chinese characterst ring sequence by cut2marks ;second ,every short sting is weighted by it s f requency in context ,and the short st rings weighted heavy are regarded as candidate words ; third ,short st rings are segmented by candidate word set and everyday words. Experiment s result s shows that the segmentation precision of this word segmentation system is aboue 1.5%,and a arge part of new words can be recognized correctly. This system is very suitable to document ret rieval and other areas.