图卷积网络与自注意机制在文本分类任务上的对比分析

摘要
参考文献
相关文章 (15)

全文: PDF (2273 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要图卷积网络近年来受到大量关注，同时自注意机制作为Transformer结构及众多预训练模型的核心之一也得到广泛运用。该文从原理上分析发现，自注意机制可视为图卷积网络的一种泛化形式，其以所有输入样本为节点，构建有向全连接图进行卷积，且节点间连边权重可学。在多个文本分类数据集上的对比实验一致显示，使用自注意机制的模型较使用图卷积网络的对照模型分类效果更佳，甚至超过了目前图卷积网络用于文本分类任务的最先进水平，并且随着数据规模的增大，两者分类效果的差距也随之扩大。这些证据表明，自注意力机制更具表达能力，在文本分类任务上能够相对图卷积网络带来分类效果的提升。

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	蒋浩泉
	张儒清
	郭嘉丰
	范意兴
	程学旗

关键词 ：图卷积网络, 自注意机制, 文本分类

Abstract：Graph Convolutional Networks has drawn much attention recently, and the self-attention mechanism has been widely applied as the core of the Transformer and many pre-trained models. We disclose that the selfattention mechanism can be seen as a generalization of Graph Convolutional Networks, in that it takes all input samples as nodes and then constructs a directed fully connected graph with learnable edge weights for convolution. Experiments show that the selfattention mechanism achieves better text classification accuracy than many state of the art Graph Convolutional Networks. Meanwhile, the performance gap of classification widens as the data size increases. These show that the self-attention mechanism is more expressive, and may surpass Graph Convolutional Networks with potential performance improvements on the task of text classification.

Key words： graph convolutional networks self-attention mechanism text classification

收稿日期: 2020-11-09

中图分类号:

TP391

基金资助:北京智源人工智能研究院项目（BAAI2019ZD0306）；
国家自然科学基金（62006218，61902381，61773362，61872338）；
中国科学院青年创新促进项目（20144310，2016102，2021100）；
国家重点研发计划（2016QY02D0405）；
联想中科院联合实验室青年科学家项目；
王宽诚教育基金会项目；
重庆市基础科学与前沿技术研究专项项目（重点）（cstc2017jcjyBX0059）

作者简介: 蒋浩泉（1997—），硕士，主要研究领域为自然语言处理。E-mail: jianghaoqaun18g@ict.ac.cn 张儒清（1994—），通信作者，博士，助理研究员，主要研究领域为自然语言处理。E-mail: zhangruqing@ict.ac.cn 郭嘉丰（1980—），博士，研究员，主要研究领域为数据挖掘、信息检索。E-mail: guojiafeng@ict.ac.cn

链接本文:

http://jcip.cipsc.org.cn/CN/Y2021/V35/I12/84

［1］Goodfellow I, Bengio Y, Courville A. Deep learning［M/OL］. Massachusetts: MIT Press, 2016.
［2］Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition［J］. Proceedings of the IEEE, 1998, 86(11): 2278-2324.
［3］Kim Y. Convolutional neural networks for sentence classification［C］//Proceedings of the Conference on Empirical Methods in Natural Language Processing. Doha, Qatar: Association for Computational Linguistics, 2014: 1746-1751.
［4］Hochreiter S, Schmidhuber J. Long short-term memory［J］. Neural Comput,1997, 9(8): 1735-1780.
［5］Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate［C］//Proceedings of the 3rd International Conference on Learning Representations, Conference Track Proceedings, 2015.
［6］Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need［M］//Proceedings of the Advances in Neural Information Processing Systems 30. Curran Associates, Inc., 2017: 5998-6008.
［7］Devlin J, Chang M W, Lee K, et al.BERT: pre-training of deep bidirectional transformers for language understanding［C］//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 4171-4186.
［8］Wu Z, Pan S, Chen F, et al. A comprehensive survey on graph neural networks［J］. IEEE transactions on neural networks and learning systems, 2020, 32(1): 4-24.
［9］Bruna J, Zaremba W, Szlam A, et al.Spectral networks and locally connected networks on graphs［C］//Proceedings of the 2nd International Conference on Learning Representations, Conference Track Proceedings, 2014.
［10］Henaff M, Bruna J, Lecun Y. Deep convolutional networks on graph-structured data［J/OL］. CoRR, 2015, abs/1506.05163.
［11］Defferrard M, Bresson X, Vanderghe-ynst P. Convolutional neural networks on graphs with fast localized spectral filtering［C］//Proceedings of the Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems Spain, 2016: 3837-3845.
［12］Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks［C］//Proceedings of the 5th International Conference on Learning Representations, Conference Track Proceedings. OpenReview.net, 2017: 1-14.
［13］〖JP+2〗Marcheggiani D, titov I. Encoding sentences with graph convolutional networks for semantic role labeling［C］//Proceedings of the Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2017: 1506-1515.〖JP〗
［14］Yao L, Mao C, Luo Y. Graph convolutional networks for text classification［C］//Proceedings of the 33rd AAAI Conference on Artificial Intelligence, AAAI Press, 2019: 7370-7377.〖JP〗
［15］Huang L, Ma D, Li S, et al.Text level graph neural network for text classification［C］//Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Association for Computational Linguistics, 2019: 3442-3448.
［16］Bastings J, Titov I, Aziz W, et al. Graph convolutional encoders for syntax-aware neural machine translation［C］//Proceedings of the Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2017: 1957-1967.
［17］Marcheggiani D, Bastings J, Titov I. Exploiting semantics in neural machine translation with graph convolutional networks［C］//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, 2018: 486-492.
［18］Fernandes P, Allamanis M, Brockschmidt M. Structured neural summarization［C］//Proceedings of the 7th International Conference on Learning Representations, New Orleans, LA, USA, 2019.
［19］Ding M, Zhou C, Chen Q, et al.Cognitive graph for multi-hop reading comprehension at scale［C］//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: As-sociation for Computational Linguistics, 2019: 2694-2703.
［20］Zhang S, Tong H, Xu J, et al. Graph convolutional networks: a comprehensive review［J］. Computational Social Networks, 2019, 6(1): 11.
［21］Yang Z, Yang D, Dyer C, et al.Hierarchical attention networks for document classification［C］//Proceedings of the Conference on the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, 2016: 1480-1489.
［22］Ying H, Zhuang F, Zhang F, et al. Sequential recommender system based on hierarchical attention networks［C］//Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI 2018: 3926-3932.
［23］〖JP〗Shen T, Zhou T, Long G, et al.Disan: Directional self-attention network for rnn/cnn-free language understanding［C］//Proceedings of the 32nd AAAI Conference on Artificial Intelligence, the 30th innovative Applications of Artificial Intelligence, and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, Louisiana, USA, 2018: 5446-5455.〖JP〗
［24］Kiela D, Wang C, Cho K. Dynamic meta-embeddings for improved sentence representations［C］//Proceedings of the Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2018: 1466-1477.
［25］Chaudharis, Polatkang, Ramanath R, et al. An attentive survey of attention models［J］. ACM Transactions on Intelligent Systems and Tech-nology (TIST), 2021, 12(5): 1-32.
［26］Velickovic P, Cucurull G, Casanova A, et al.Graph attention networks［C］//Proceedings of the 6th International Conference on Learning Representations, Canada, 2018.
［27］Zhang J, Shi X, Xie J, et al. gaan: gated attention networks for learning on large and spatio temporal graphs［C］//Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, AUAI Press, 2018: 339-349.
［28］Lee J B, Rossi R A, Kim S, et al. Attention models in graphs: a survey［J］. ACM Trans. Knowl. Discov. Data, 2019, 13(6): 62.
［29］Pennington J, Socher R, Manning C D. Glove: global vectors for word representation［C］//Proceedings of the Conference on Empirical Methods in Natural Language Processing, ACL, 2014: 1532-1543.
［30］Kingma D P, Ba J. Adam: a method for stochastic optimization［C］//Proceedings of the 3rd International Conference on Learning Representations, Conference Track Proceedings, 2015.