基于门控卷积机制与层次注意力机制的多语义词向量计算方法

摘要
参考文献
相关文章 (12)

全文: PDF (6422 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要现有的将词映射为单一向量的方法没有考虑词的多义性,从而会引发歧义问题;映射为多个向量或高斯分布的方法虽然考虑了词的多义性,但或多或少没能有效利用词序、句法结构和词间距离等信息对词在某一固定语境中语义表达的影响。综合考虑以上存在的问题,该文提出了一种基于非残差块封装的门控卷积机制加以层次注意力机制的方法,分别在所选取语境窗口中词的子语义层、合成语义层获得非对称语境窗口下目标单词的合成语义向量以预测目标单词,并按此法在给定语料上学习得到多语义词向量的计算方法。小规模语料上用该方法得到的多语义词向量,在词类比任务的语义类比上相比于基线方法准确率最高可提升1.42%;在WordSim353、MC、RG、RW等计算单词相似度任务的数据集上相比于基线方法能够达到平均2.11的性能提升,最高可到5.47。在语言建模实验上,该方法的语言模型性能相比于其他预测目标单词的方法也有显著提升。

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	柳杨
	吉立新
	黄瑞阳
	朱宇航
	李星

关键词 ：多语义词向量, 层次注意力, 门控卷积

Abstract：The existing methods (mapping a word to a single vector) do not consider the problem of polysemy, which may cause the problem of ambiguity; Rather than mapping a word to multiple vectors, this paper proposes a computing method of multi-sense word embedding by: 1) fusing hierarchical attention mechanism with non-residual encapsulated gated convolution mechanism in the sub-sense layer and synthetic sense layer of the words in the selected context window, and 2) obtains the synthetic sense embedding of the target word under the asymmetric window to predict the target word. On small-scale corpus, the proposed multi-sense word embedding achieves at most 1.42% increase in the accuracy of the word analogy task, an average 2.11% (up to 5.47%) improvement in the word similarity tasks including WordSim353, MC, RG, and RW. In addition, this method also significantly improves the performance of the language modeling compared with other methods predicting target words.

Key words： Multi-sense word embedding hierarchical attention gated convolution

收稿日期: 2017-08-31

中图分类号:

TP391.1

基金资助:国家自然科学基金(61601513)

作者简介: 柳杨(1994—),硕士研究生,主要研究领域为自然语言处理。E-mail:fbdst456@126.com
吉立新(1969—),博士,研究员,主要研究领域为通信与信息系统。E-mail:jlxndsc@139.com
黄瑞阳(1986—),博士,助理研究员,主要研究领域为文本挖掘、图挖掘。E-mail:277433109@qq.com

链接本文:

http://jcip.cipsc.org.cn/CN/Y2018/V32/I7/1

[1] Mikolov T, Le Q V, Sutskever I. Exploiting similarities among languages for machine translation [J]. arXiv preprint arXiv: 1309. 4168, 2013.
[2] Mikolov T, Chen K, Corrado G, et al. Efficient estimation of word representations in vector space [J]. arXiv preprint arXiv: 1301. 3781, 2013.
[3] Mikolov T, Sutskever I, Chen K, et al. Distributed representations of words and phrases and their compositionality[J]. Advances in neural information processing systems, 2013, 26: 3111-3119.
[4] Mikolov T, Yih W, Zweig G. Linguistic regularities in continuous space word representations[C]//Proceedings of the hlt-Naacl. 2013(13): 746-751.
[5] Pennington J, Socher R, Manning C D. Glove: Global vectors for word representation [C]//Proceedings of the EMNLP, 2014(14): 1532-1543.
[6] Ji S, Yun H, Yanardag P, et al. Wordrank: Learning word embeddings via robust ranking[J]. arXiv preprint arXiv: 1506. 02761, 2015.
[7] Joulin A, Grave E, Bojanowski P, et al. Bag of tricks for efficient text classification[J]. arXiv preprint arXiv: 1607. 01759, 2016.
[8] Bojanowski P, Grave E, Joulin A, et al. Enriching word vectors with subword information[J]. arXiv preprint arXiv: 1607. 04606, 2016.
[9] Vilnis L, McCallum A. Word representations via gaussian embedding[J]. arXiv preprint arXiv: 1412. 6623, 2014.
[10] Chen X, Qiu X, Jiang J, et al. Gaussian mixture embeddings for multiple word prototypes[J]. arXiv preprint arXiv: 1511. 06246, 2015.
[11] Huang E H, Socher R, Manning C D, et al. Improving word representations via global context and multiple word prototypes[C]//Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers Volume 1. Association for Computational Linguistics, 2012: 873-882.
[12] Neelakantan A, Shankar J, Passos A, et al. Efficient non-parametric estimation of multiple embeddings per word in vector space[J]. arXiv preprint arXiv: 1504. 06654, 2015.
[13] NiuYilin, Xie Ruobing, Liu Zhiyuan, et al. Improved Word Representation Learning with Sememes[C]//Proeedings of The 55th Annual Meeting of the Association for Computational Linguistics (ACL'17). Long paper. Canada, 2017(to be published).
[14] Dauphin Y N, Fan A, Auli M, et al. Language modeling with gated convolutional networks[J]. arXiv preprint arXiv: 1612. 08083, 2016.
[15] Gehring J, Auli M, Grangier D, et al. Convolutional sequence to sequence learning[J]. arXiv preprint arXiv: 1705. 03122, 2017.
[16] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.
[17] Sutskever I, Martens J, Dahl G, et al. On the importance of initialization and momentum in deep learning[C]//Proceedings of the International Conference on Machine Learning, 2013: 1139-1147.
[18] Pascanu R, Mikolov T, Bengio Y. On the difficulty of training recurrent neural networks[C]//Proceedings of International Conference on Machine Learning. 2013: 1310-1318.
[19] Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the International Conference on Machine Learning, 2015: 448-456.
[20] Finkelstein L, Gabrilovich E, Matias Y, et al. Placing search in context: The concept revisited[C]//Proceedings of the 10th International Conference on World Wide Web. ACM, 2001: 406-414.
[21] Miller G A, Charles W G. Contextual correlates of semantic similarity[J]. Language and Cognitive Processes, 1991, 6(1): 1-28.
[22] Rubenstein H, Goodenough J B. Contextual correlates of synonymy[J]. Communications of the ACM, 1965, 8(10): 627-633.
[23] Luong T, Socher R, Manning C D. Better word representations with recursive neural networks for morphology[C]//Proceedings of the CoNLL, 2013: 104-113. [24] Marcus M P, Marcinkiewicz M A, Santorini B. Building a large annotated corpus of English: The Penn Treebank[J]. Computational Linguistics, 1993, 19(2): 313-330.
[25] Levy O, Goldberg Y, Ramat-Gan I. Linguistic regularities in sparse and explicit word representations[C]//Proceedings of the CoNLL, 2014: 171-180.