基于层次混合注意力机制的文本分类模型

摘要
参考文献
相关文章 (15)

全文: PDF (3131 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要文本分类是自然语言处理领域的核心任务之一,深度学习的发展给文本分类带来更广阔的发展前景。针对当前基于深度学习的文本分类方法在长文本分类中的优势和不足,该文提出一种文本分类模型,在层次模型基础上引入混合注意力机制来关注文本中的重要部分。首先,按照文档的层次结构分别对句子和文档进行编码;其次,在每个层级分别使用注意力机制。句编码时在全局目标向量基础上同时利用最大池化提取句子特定的目标向量,使编码出的文档向量具有更加明显的类别特征,能够更好地关注到每个文本最具区别性的语义特征。最后,根据构建的文档表示对文档分类。在公开数据集和行业数据集上的实验结果表明,该模型对具有层次结构的长文本具有更优的分类性能。

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	孙新
	唐正
	赵永妍
	张颖捷

关键词 ：文本分类, 深度学习, 注意力机制

Abstract：Text classification is one of the core tasks in the field of natural language processing. To address the long text sequence, we propose the Hierarchical Networks with Mixed Attention (HMAN) model for text classification to capture the important parts of the text based on the hierarchical model. First, the sentences and documents are encoded according to the hierarchical structure of documents, and attention mechanism is applied at each level. Then, global target vectors, sentence specific target vectors are extracted by max-pooling to encode the document vectors. Finally, documents are classified according to the constructed document representation. Experimental results on the open datasets and industry text datasets show that the model has better classification performance, especially for long text with hierarchical structure.

Key words： text classification deep learning attention mechanism

收稿日期: 2019-12-18

中图分类号:

TP391

基金资助:国家重点研发计划项目(2017YFB0803300)

作者简介: 孙新(1975—),博士,副教授,主要研究领域为人工智能、机器学习。E-mail:sunxin@bit.edu.cn
唐正(1994—),硕士研究生,主要研究领域为机器学习、智能问答。E-mail:2120171060@bit.edu.cn
赵永妍(1997—),硕士研究生,主要研究领域为文本分类。E-mail:3220190927@bit.edu.cn

链接本文:

http://jcip.cipsc.org.cn/CN/Y2021/V35/I2/69

[1] Lewis D D. Naive (Bayes) at forty: The independence assumption in information retrieval [M]. Berlin Heidelberg: Springer, 1998: 4-15.
[2] Cover T, Hart P. Nearest neighbor pattern classification [J]. IEEE Transactions on Information Theory, 2003, 13(1): 21-27.
[3] Cortes C,Vapnik V. Support-vector networks [J]. Machine Learning, 1995, 20(3): 273-297.
[4] 周飞燕, 金林鹏, 董军. 卷积神经网络研究综述[J]. 计算机学报, 2017(6): 1229-1251.
[5] Yoon Kim . Convolutional neural networks for sentence classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 2014: 1746-1751.
[6] Liu Pengfei, Qiu Xipeng, Huang Xuanjing. Recurrent neural network for text classification with multi-task learning[C]//Proceedings of the International Joint Conferences on Artificial Intelligence, 2016: 2873-2879.
[7] Tang Duyu, Qin Bing, Liu Ting. Document modeling with gated recurrent neural network for sentiment classification[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 2015: 1422-1432.
[8] Yang Zichao, Yang Diyi, Dyer Chris. Hierarchical attention networks for document classification[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016: 1480-1489.
[9] 程艳,叶子铭,王明文等.融合卷积神经网络与层次化注意力网络的中文文本情感倾向性分析[J].中文信息学报,2019,33(1): 133-142.
[10] XiaoY , Cho K . Efficient Character-level document classification by combining convolution and recurrent layers [J/OL]. arXiv preprint arXiv: 1602.00367, 2016.
[11] Conneau A , Schwenk H , Barrault, Loc, et al. Very deep convolutional networks for text classification[C]//Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 2016: 1107-1116.
[12] Rie Johnson, Tong Zhang. Deep pyramid convolutional neural networks for text categorization[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017: 562-570.
[13] Xu Jiacheng, Chen Danlu, Qiu Xipeng, et al. Cached long short term memory neural networks for document-level sentiment classification[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2016: 1660-1669.
[14] Mnih V , Heess N , Graves A , et al. Recurrent models of visual attention[J]. Advances in Neural Information Processing Systems, 2014(3): 2204-2212.
[15] Bahdanau D , Cho K , Bengio Y . Neural machine translation by jointly learning to align and translate [C]//Proceedings of the 2015 International Conference on Learning Representations, 2015: 1-15.
[16] Quinlan J R. Learning efficient classification procedures and their application to chess end games[M]. Machine Learning: An Artificial Intelligence Approach, 1983(1): 463-482.
[17] McCallum A, Nigam K. A comparison of event models for naive Bayes textclassification[G]//Proceedings of AAAI-98 Workshop on Learning for Text Categorization, 1998, 752(1): 41-48.
[18] Breiman L, Friedman J, Stone J C, et al. Classification and regression trees [M]. London: Chapman and Hall/CRC.1984.
[19] Zhang X, Zhao J,LeCun Y. Character level convolutional networks for text classification[C]//Proceedings of the 2015 Conference and Workshop on Neural Information Processing Systems, 2015: 649-657.