问答系统能用准确、简洁的答案回答用户用自然语言提出的问题,很明显系统中问答对的规模是影响问答系统最终性能的主要因素。为了提高问答对的规模、充分利用互联网资源,本文提出了一种基于决策树和马尔科夫链的在互联网上自动抽取问答对的算法。先根据网页中的HTML标记把网页表示成一棵DOM树;然后利用树中每个节点的结构和文字信息,抽取相应的特征;最后将得到的节点特征通过由决策树和一阶马尔可夫链结合得出的分类模型进行分类。试验结果表明准确率达到了90.398%,召回率达到了86.032%。对大量网页抽取的结果表明该分类模型能够适应对各种各样的网页的抽取。
Abstract
Question Answering System can give users precise answer to the question presented in natural language and the major factor which influence the System’s performance is the scale of Question-Answer pairs. In order to increase the Question-Answer pair’s scale and make full use of Web Pages’ resource, in this paper we propose a method that uses decision tree and Markov model to extract Question-Answer pairs in Web Pages. The method uses DOM tree to represent a web page according to HTML tags. Then acquire features value from every DOM tree’s node. Last allow the features overpass the classification model, which created by decision tree and Markov model, to get the node’s last classification result. Experimental results show that the precision achieved 90.40% and recall achieved 86.03%. Experimental results also show that this model could extract information from all kinds of Web Pages.
关键词
人工智能 /
模式识别 /
信息抽取 /
DOM树 /
决策树 /
马尔可夫链
{{custom_keyword}} /
Key words
artificial intelligence /
pattern recognition /
information extraction /
DOM tree /
decision tree /
Markov model
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] Craven, T.C. HTML Tags as Extraction Cues for Web Page Description Construction [J]. Informing Science Journal, 2003, 6: 1-12.
[2] Kosala, R.,Bruynooghe, M.,Bussche, J.V.,et al. Information Extraction from Web Documents Based on Local Unranked Tree Automaton Inference [A]. In: Proceedingsof the Eighteenth International Joint Conference on Artificial Intelligence (IJCAI-2003) [C]. 2003.
[3] Eikvil, L.Information Extraction from World Wide Web- A Survey [R]. Technical Report 945, 1999.
[4] Reis, D., Golgher, P., Silva, A.,et al. Automatic Web News Extraction Using Tree Edit Distance [A]. In: Proceedings of International WWW Conference (WWW-2004) [C]. 2004,502-511.
[5] Yunhua Hu, Guomao Xin,et al. Title Extraction from Bodies of HTML Documents and its Application to Web Page Retrieval [A]. The 28th Annual International ACM SIGIR Conference (SIGIR’2005) [C]. August 2005.
[6] 何新贵,彭甫阳. 中文文本的关键词自动抽取和模糊分类[J]. 中文信息学报, 1999,13(1): 9-15.
[7] Breuel, T.M. Information Extraction from HTML Documents by Structural Matching. In: Proceedings of the Second International Workshop on Web Document Analysis(WDA2003), 2003.
[8] 李效东,顾毓清. 基于DOM的Web的信息提取[J]. 计算机学报,2002,(5): 526-533.
[9] J.R. Quinlan. C4.5 Programs for Machine Learning [J]. Morgan Kaufmannn Publishers San Meteo, California, 1992.
[10] 于琨,蔡智,等. 基于路径学习的信息自动抽取方法[J]. 小型微型计算机系统, 2003, (12): 2147-2149.
[11] BhnI. Web Document Clustering: A feasibility Demonstration [A]. In: Proceeding of the 21st SIGIR Conference [C]. Melbourne, Australia, 1999. 46-54.
[12] Ron Kohavi. A Study of Cross_Validation and Bootstrap for Accuracy Estimation and Model Selection [A]. In: International Joint Conference on Artificial Intelligence (IJCAI) [C]. 1995.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
国家自然科学基金资助项目(60672056);微软基金资助项目(2006120809)
{{custom_fund}}
PDF(462 KB)
