Web论坛中蕴含着丰富的信息资源,充分利用这些信息资源依赖于论坛数据抽取技术。该文解决了从Web论坛抽取什么数据和如何抽取的问题,提出了一种基于网页布局相似度的Web论坛数据抽取方法,有效弥补了目前方法的自动化程度低,或准确率低的不足。该方法充分利用Web论坛网页布局结构上的特点,采用分级处理的方式,先识别出主题信息块、再利用待抽取数据的统计规律在主题信息块中完成抽取,整个过程不需要任何人工干预。实验结果表明,新方法对不同的BBS站点有很好的通用性,且具有较高的准确率和召回率。
Abstract
Web forums contain a wealth of information resources. Making full use of these information resources relies on web forums data extraction technology. This paper solves the problems of what data should be extracted and how to extract from web forums by the proposed method based on the similarity of page layout. The method can effectively avoid the disadvantages of current methods at low degree of automation or low accuracy. The method firstly recognizes the topic block by making full use of the special layout of the web forum pages, then extract data using rules from the topic block. Experimental results show that this method performs well in adjustability, precision and recall.
Key wordscomputer application; Chinese information processing;Web forum; data extraction; similarity
关键词
计算机应用 /
中文信息处理 /
Web论坛 /
数据抽取 /
相似度
{{custom_keyword}} /
Key words
computer application /
Chinese information processing /
Web forum /
data extraction /
similarity
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] 薛玮.网络舆情信息挖掘系统的研究[D].北京:北京交通大学,2008.
[2] 姚晓娜.BBS热点话题挖掘与观点分析[D].大连:大连海事大学,2008.
[3] J. Zhang, M. S. Ackerman, and L. Adamic. Expertise networks in online communities: structure and algorithms[C]//Proc. 16th WWW, Banff, Canada, May 2007. 2007:221-230.
[4] I. Muslea, S. Minton, C. Knoblock. A Hierarchical Approach to Wrapper Induction[C]//Third International Conference on Autonomous Agents, (Agents’99), Seattle, May 1999.
[5] S. Soderland. Learning Information Extraction Rules for Semistructured and Free Text[J]. Machine Learning, 1999.
[6] Liu B., Grossman R., Zhai Y. Mining Data Records in Web Pages [C]//KDD 2003: 601-606.
[7] Z. Yanhong and L. Bing, Web Data Extraction Based on Partial Tree Alignment[C]//Proceedings of the ACM, 2005: 76-85.
[8] Liu, B. and Zhai, Y., NET - A System for Extracting Web Data from Flat and Nested Data Records[C]//WISE 2005, 2005: 487-495.
[9] Justin Park and Denilson Barbosa.Adaptive Record Extraction From Web Pages[C]//WWW 2007.
[10] Gusfield, D. Algorithms on strings, tree, and sequence[M]. Cambridge. 1997.
[11] 韩先培,刘康,赵军.基于布局特征与语言特征的网页主要内容块发现[J].中文信息学报,2008,22(1):15-21.
[12] 瞿有利,于浩,徐国伟,等.Web页面信息块的自动分割[J].中文信息学报,2003,18(1):6-13.
[13] 李保利,陈玉忠,俞士汶.信息抽取研究综述[D].北京:北京大学计算机科学与技术系计算语言研究所,2003.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
国家863计划资助项目(2007AA01Z439);信息工程大学学位论文创新基金资助项目(BSLWCX200802)
{{custom_fund}}
PDF(1913 KB)
