本文旨在研究如何基于小世界模型进行文本分割,确定片段主题,进而总结全文的中心主题,使文本的主题脉络呈现出来。为此首先证明由文本形成的词汇共现图呈现短路径,高聚集度的特性,说明小世界结构存在于文本中;然后依据小世界结构将词汇共现图划分为“簇”,通过计算“簇”在文本中所占的密度比重识别片段边界,使“簇”与片段对应起来;最后利用短路径,高聚集度的特性提取图“簇”的主题词,采取背景词汇聚类及主题词联想的方式将主题词扩充到待分析文本之外,尝试挖掘隐藏于字词表面之下的文本内涵。虽然国际上已有很多关于小世界结构及基于其上的应用研究,但利用小世界特性进行主题分析还是一个崭新的课题。实验表明,本文所给方法的结果明显好于其他方法,说明可以为下一步文本推理的工作提供有价值的预处理。
Abstract
The paper aims to perform topic spotting of segments based on text segmentation using small world structure. Main topic of the whole text is generalized and the skeleton of text shows itself. It is explained that the term co-occurrence graph of text is highly clustered and has short path length, which proves that texts have small world structure. Clusters in the small world structure are detected. The density of each cluster is computed to discover the boundary of a segment. Topic words are extracted from clusters of the graph. Words which are not distinctly in the analyzed text can be included to express the topics with the help of word clustering of background and topic words association .The signification behind the words are attempted to dig out. Although much research on applications of small world structure, it is a new task to analyze texts with the characteristics of small world. The experiments tell that the result of tests is far better than that of other methods. Valuable pre-processing is provided for next work of text reasoning.
关键词
计算机应用 /
中文信息处理 /
主题分析 /
小世界模型 /
文本分割 /
词汇聚类
{{custom_keyword}} /
Key words
computer application /
Chinese information processing /
topic analysis /
small world model (SWD) /
text segmentation /
words clustering
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] Ath. Kehagias, A. Nicolaou, P. Fragkou and V. Petridis. Text Segmentation by Product Partition Models and Dynamic Programming [J]. Mathematical and Computer Modeling, 2004. 39: 209-217.
[2] Gina-Anne Levow. Prosody-based topic segmentation for mandarin broadcast news [A]. In: Proceedings of HLT-NAACL [C]. 2004.
[3] Ferret, Olivier. Using collocations for topic segmentation and link detection [A]. In: Proceedings of COLING [C]. Taipei. 260-266.
[4] Hang Li and Kenji Yamanishi. Topic Analysis Using a Finite Mixture Model [J]. Information Processing & Management, 2003, 39(4): 521-541.
[5] Brants, T.; Chen, F. R.; Farahat, A. O. Arabic document topic analysis [A]. In: LREC-2002 Workshop on Arabic Language Resources and Evaluation [C]. Las Palmas; Spain, 2002.
[6] D.M. Blei, A.Y. Ng, and M.I. Jordan. Latent Dirichlet allocation [J]. Journal of Machine Learning Research, 2003(3):993-1022.
[7] Steyvers, M. & Griffiths, T. Probabilistic topic models. In: T. Landauer, D. S. McNamara, S. Dennis, & W. Kintsch (Eds.), handbook of Latent Semantic Analysis [M]. Hillsdale, NJ: Erlbaum. 2007.
[8] 索红光, 刘玉树,曹淑英.一种基于词汇链的关键词抽取方法[J]. 中文信息学报, 2006,20(6): 27-32.
[9] Ferrer-i-Cancho, R. and Sole, R. V. The small world of human language [A]. In: Proceedings of the Royal Society of London. Series B, Biological Sciences [C]. 2001. 268(1482): 2261-2265.
[10] Matsuo, Y.; Ohsawa, Y.; and Ishizuka, M. A docu-ment as a small world [A]. In: Proceedings the 5th World Multi-Conference on Systemics, Cybenetics and Infomatics [C]. 2001, 8: 410-414.
[11] D.Watts and S. Strogatz. Collective dynamics of small-world networks [J]. Nature, 1998, 393: 440-442.
[12] Yutaka Matsuo: Clustering using Small World Structure [A]. In: Proc. 6th Int’l Conf. on Knowledge-based Intelligent Information Engineering Systems & Applied Technologies(KES2002) [C]. IOS Press/Ohmsha (ISSN:0922-6389), Crema, Italy, 2002, 1252-1256.
[13] Yutaka Matsuo, Yukio Ohsawa and Mitsuru Ishizuka: KeyWorld: Extracting Keywords in a Document as a Small World [A]. In: DS-2001 [C]. 2001, 271-281.
[14] D. Beeferman, A. Berger, J. Lafferty. Statistical Models for Text Segmentation [J]. In: Machine Learning, 1999,34,1-34.
[15] L. Pevzner and M. Hearst. A critique and improvement of an evaluation metric for text segmentation [J]. Computational Linguistics. 2002,28(1): 19-36.
[16] Thorsten Brants, Francine Chen, Ioannis Tsochantaridis. Topic-based document segmentation with probabilistic latent semantic analysis [A]. In: Proceedings of the eleventh international Conference on Information and knowledge management [C]. McLean, Virginia, USA.2002.211-218.
[17] F. Y. Y. Choi, P. Wiemer-Hastings, and J. Moore. Latent semantic analysis for text segmentation [A]. In: Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing [C]. 2001.109~117.
[18] 石晶,戴国忠. 基于PLSA模型的文本分割[J]. 计算机研究与发展,2007, 44(2): 242-248.
[19] Liu Y, Ciliax BJ, Borges K, Dasigi V, Ram A, Navathe SB, Dingledine R. Comparison of two schemes for automatic keyword extraction from MEDLINE for functional gene clustering [A]. In: Procceedings of the 2004 IEEE Computational Systems Bioinformatics Conference (CSB’04) [C]. 2004. 394-404.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
国家自然科学基金资助项目(60305006)
{{custom_fund}}
PDF(314 KB)
