随着互联网的发展,社会媒体已经逐渐发展成为信息交流的重要载体。该文针对社会媒体文本的领域分布广、口语化程度高等特征,提出一种面向社会媒体的开放领域新词发现算法。此算法所有步骤均为线性时间复杂度,并且在分析过程中有效降低了内存的使用,从而能够实时处理社会媒体所产生的大规模数据。在6.6 GB 社会媒体文本语料中的新词发现准确率达到了87.2%,在普通计算机上新词发现速度可达2.6 MB/s。与传统算法相比,该算法在社会媒体领域的大规模语料中速度及精度上均有较好的效果。

With the development of Internet, social media has become an important channel for information transmission. Focused on characteristics of the informal language in various domains inherent in social media, this paper proposes a social media-oriented open domain new word detection method. This approach can be executed in linear time complexity with a reduced memory usage, which enables real time processing large size data produced by social media. The experiment on a 6.6GB social media corpus reveal a processing speed of 2.6MB/s in normal PC, as well as 87.2% precision.