句子排序是自然语言处理中的重要任务之一,其应用包括多文档摘要、问答和文本生成。不当的句子排序会产生逻辑不通顺的文本,使得文本的可读性降低。该文采用在中英文上大规模使用的深度学习方法,同时结合朝鲜语词语形态变化丰富的特点,提出了一种基于子词级别词向量和指针网络的朝鲜语句子排序模型,其目的是解决传统方法无法挖掘深层语义信息的问题。该文提出基于形态素拆分的词向量训练方法(MorV),同时对比子词n元词向量训练方法(SG),得到朝鲜语词向量;采用了两种句向量方法: 基于卷积神经网络(CNN)、基于长短时记忆网络(LSTM),结合指针网络分别进行实验。结果表明采用MorV和LSTM的句向量结合方法可以更好地捕获句子间的语义逻辑关系,提升句子排序的效果。
Abstract
Sentence ranking is one of the core technologies in natural language processing, with wide applications in multi-document summarization, question answering and text generation. Considering the rich morphological changes of Korean words, this paper a puts forward a Korean sentence ordering model based on the sub-word level vector and pointer network. A morpheme split based word vector training method (MorV) is presented, and the Korean word vector is obtained by comparing the sub word n-gram word vector training method (SG). Two sentence vector methods, i.e. convolution neural network (CNN) and long-term memory network (LSTM), are explored in the pointer network. The results show that the combination of MorV and LSTM can better capture the semantic logic relationship between sentences to improve the sentence ordering.
关键词
词向量 /
形态素拆分 /
指针网络 /
句子排序
{{custom_keyword}} /
Key words
word vector /
morpheme split /
pointer network /
sentence ordering
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] 徐超, 毕玉德. 面向自然语言处理的韩国语隐喻知识库构建研究[C]. 中国计算语言学研究前沿进展(2009-2011). 清华大学出版社, 2011, 296-301.
[2] Bollegala, Danushka, Naoaki Okazaki, et al. A machine learning approach to sentence ordering for multidocument summarization and its evaluation[C]//Proceedings of the International Conference on Natural Language. Springer, Berlin, Heidelberg, 2005: 624-635.
[3] Ziqiang Cao, Furu Wei, Dong Li, et al. Ranking with recursive neural networks and its application to multi-document summarization[C]//Proceedings of National Conference on Artificial Intelligence, AAAI Press, Austin, Texas, USA, 2015: 2153-2159.
[4] 栾克鑫,杜新凯,孙承杰,等. 基于注意力机制的句子排序方法[J]. 中文信息学报,2018,32(01): 123-130.
[5] 康世泽,马宏,黄瑞阳. 一种基于神经网络模型的句子排序方法[J]. 中文信息学报,2016,30(05): 195-202.
[6] Logeswaran, Lajanugen, Honglak Lee, et al. Sentence ordering using recurrent neural networks[J/OL]. arXiv prepre arXiv: 1611.02634:1611.
[7] Jingjing Gong, Xinchi Chen, Xipeng Qiu, et al. End-to-End neural sentence ordering using pointer network[J/OL]. arXiv prepre arXiv: 1607. 06952 2016.
[8] Vinyals Oriol, Meire Fortunato, Navdeep Jaitly. Pointer networks[C]//Proceedings of the the 29th International Conference on Neural Information Processing Systems, 2015: 2692-2700.
[9] Nallapati, Ramesh, Bowen Zhou, et al. Abstractive text summarization using sequence-to-sequence rnns and beyond[J/OL]. arXiv preprint arXiv:1602. 06023,2016.
[10] 金国哲,崔荣一. 一种新的朝鲜语词性标注方法[J]. 中文信息学报,2018,32(10): 53-58,68.
[11] Song, Hyun Je, Seong Bae Park. Korean morphological analysis with tied sequence-to-sequence multi-task model[C]//Proceedings of the Conference on Empirical Methods in Natural Language and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China,2019: 1436-1441.
[12] Nallapati, Ramesh, Bowen Zhou, et al. Abstractive text summarization using sequence-to-sequence rnns and beyond[J/OL]. arXiv preprint arXiv: 1602. 06023,2016.
[13] Sungjoon Park, Jeongmin Byun, Sion Baek, et al. Subword-level word vector representations for Korean[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Melbourne, Australia, 2018: 2429-2438.
[14] Piotr Bojanowski, Edouard Grave, Armand Joulin, et al. Enriching word vectors with subword information[J]. Transactions of the Association for Computational Linguistics, 2017, 5: 135-146.
[15] Tomas Mikolov, Kai Chen, Greg Corrado, et al. Efficient estimation of word representations in vector space[J/OL]. arXiv preprint arXiv: 1301. 3781,2013.
[16] Daniel Cer, Yinfei Yang, Shengyi Kong, et al. Universal sentence encoder arXiv preprint arXiv: 1803. 11175,2018.
[17] Simard, Patrice Y, David Steinkraus, et al. Best practices for convolutional neural networks applied to visual document analysis[C]//Proceedings of 7th International Conference on Document Analysis and Recognition. IEEE, Edinburgh, UK, 2003: 958-963.
[18] Alex Graves. Long short-term memory[J]. Supervised Sequence Labelling with Recurrent Neural Networks, Springer,Berlin, Heidelberg, 2012: 37-45.
[19] John Duchi, Elad Hazan, Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization[J]. Journal of Machine Learning Research, MIT Press, 2011,12(61): 2121-2159.
[20] Turian, Joseph, Lev Ratinov, et al. Word representations: A simple and general method for semi-supervised learning[C]//Proceedings of the the 48th Annual Meeting of the Association for Computational Linguistics,Association for Computational Linguistics, Uppsala Sweden, 2010: 384-394.
[21] Yoon Kim, Yacine Jernite, David Sontag, et al. Character-aware neural language models[C]//Proceedings of the 30th AAAI Conference on Artificial Intelligence. Phoenix, Arizona USA, 2016: 2741-2749.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
PDF(2460 KB)
