垂直检索系统中主题爬虫的性能对整个系统至关重要。在设计主题爬虫时需要解决两个问题一是计算当前页面与给定主题的相关度, 二是计算待爬取URLs的访问优先级。对第一个问题,给出利用页面的主题文本块和相关链接块的相关度计算方法; 对第二个问题, 给出基于主题上下文和四种不同的粒度(即站点级、页面级、块级和链接级)的优先级计算方法。在此基础上, 提出基于上述方法的主题爬取算法。实验证明, 新算法在不增加时间复杂度的前提下, 在查准率和信息量总和方面明显优于其他三种经典的爬取算法。
Abstract
The performance of the focused crawler is crucial to a vertical search engine. Two scientific computation issues to be addressed in the design of focused crawlers are(1) how to compute the relevance of a current visited Web page to a given topic, (2) how to compute the priorities of unvisited URLs in the queue. For the first issue, this paper describes the calculation of the relevance of a page to the topic based on the page's topical text blocks and related link blocks. For the second one, a novel approach is proposed to prioritize these unvisited URLs by hierarchical topic context of four different granularities, i.e. site level, page level, block level and link level. Finally, a new focused crawling algorithm is presented. Experiments show that the new algorithm is more effective than three traditional algorithms in terms of precision rate and information amount without increasing time complexity.
Key words computer application; Chinese information processing; focused crawling; URLs priority computation; page segmentation; relevance computation
关键词
计算机应用 /
中文信息处理 /
主题爬取 /
优先级计算 /
网页分块 /
相关度计算
{{custom_keyword}} /
Key words
computer application /
Chinese information processing /
focused crawling /
URLs priority computation /
page segmentation /
relevance computation
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] P. DeBra, G. Houben, Y. Kornatzky, R. Post. Information Retrieval in Distributed Hypertexts[C]//Proceedings of the 4th RIAO Conference, New York, 1994: 481-491.
[2] M. Hersovici, M. Jacovi, Y. Maarek et al. The Shark-Search Algorithm-an Application: Tailored Web Site Mapping[C]//Proceedings of the Seventh International World Wide Web Conference (WWW’98), Australia, 1998: 317-326.
[3] M. Chau, H. Chen. Comparison of Three Vertical Search Spiders[J]. IEEE Computer, 2003, 26(5): 56-62.
[4] J. Cho, H. Garcia-Molina, L. Page. Efficient Crawling Through URL Ordering[C]//Proceedings of the 7th International World Wide Web Conference, Australia 1998: 161-172.
[5] F. Menczer, G. Pant et al.. Topical Web Crawlers: Evaluating Adaptive Algorithms[J]. ACM Transaction on Internet Technology, 2004, 4(4): 378-419.
[6] P. Srinivasan, F. Menczer, G. Pant. A General Evaluation Framework for Topical Crawlers[J]. Information Retrival, 2005, 8(3): 417-447.
[7] Nick Craswell and David Hawking. Oveniew of the TEEC 2003 Web Track [OL]. http://trec.nist.gov/pubs/trec12/papers/WEB.OVERVIEW.pdf.
[8] Zh.M. Chen, J. Ma et al. An Effective Relevance Predation Algorithm based on Hierarchical Taxonomy for Focused Crawling[C]//Proceeding of the 2008 Asia Information Retrieval Symposium, 2008: 623-629.
[9] S. Chakrabarti, K. Punera, M. Subramanyam. Accelerated Focused Crawling Through Online Relevance Feedback[C]//Proceedings of the 11th International World Wide Web Conference (Hawaii 2002): 148-159.
[10] Open directory project [OL]. http://dmoz.org/.
[11] B. D. Davison. Topical locality in the Web[C]//Proceedings of the 23rd Annual International Conference on Research and Development in Information Retrieval (SIGIR 2000), Greece, July, 2000: 272-279.
[12] S. Chakrabarti, M. Joshi M. et al.. The Structure of Broad Topics on the Web[C]//Proceedings of the 11th international Conference on World Wide Web (WWW’02), ACM, 2002: 251-262.
[13] D. Cai, Sh. Yu, J.R. Wen and W.Y. Ma. VIPS: a Vision based Page Segmentation Algorithm [R]. Microsoft Technical Report, Miscrosoft, 2003.
[14] CWT200G (Chinese Web Test collection with 200 GB web pages) [OL]. Available on: http://www.cwirf.org/SharedRes/DataSet/CWT200g/CWT200g_number.
[15] B. Pinkerton. Finding What People Want: Experiences with the Web Crawler [C]//Proceedings of the 2th International World Wide Web Conference, (www94), USA, 1994.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
高等学校博士学科点专项科研基金项目(20070422107);山东省科技攻关项目(2007GG10001002);海南省自然科学基金项目(80546)
{{custom_fund}}
PDF(595 KB)
