本文提出了一种快速的高频字串提取和统计方法。使用Hash技术,该方法不需要词典,也不需要语料库的训练,不进行分词操作,依靠统计信息,提取高频字串。用语言学知识进行前缀后缀等处理后,得到的高频字串可以作为未登录词处理、歧义消解和加权处理等的辅助信息。实验显示了该方法速度较快且不受文章本身的限制,在处理小说等真实文本时体现了较高的可用性。

In this paper we describe a fast high-frequency strings extracting algorithm. Our approach uses HASH technology to avoid relying on corpus and word segmentation. To extract the high frequency strings , we only use statistics information. After processing the prefixes and suffixes , the high frequency strings we get can be the supplement knowledge for the un-login words processing , word disambiguation and word weighting. The experimental results show that it has a high speed and can work on arbitrary texts. Our method has good effect when processing novels and other real texts.