由于预训练模型对于长度的限制,长文本机器阅读理解任务必须把文章分成多个块放到预训练模型中提取答案。现有循环分块模型存在分块内部信息提取不合理、分块之间语义传输不充分的问题。针对这些问题,该文提出了基于关键词的长文本机器阅读理解循环分块模型。在对文章和问题进行词嵌入时引入外部知识库,得到丰富的文章词向量表示,在此基础上结合文章中的关键词通过强化学习策略得到更加灵活的文章分块,随后通过平衡参数得到文章最佳答案。在CoQA、QuAC和TriviaQA数据集上所提模型与BERT-LARGE模型和循环分块模型相比较F1值分别提高了 5.1和 4.5个百分点, 3.9和 3.3个百分点,3.9和 2.9个百分点。实验结果表明,该文所提模型对长文本机器阅读理解的综合效果得到有效提升,F1值均优于对比模型。
Abstract
Due to the limitation of pre-training model on length, long-text machine reading comprehension task usually divides texts into several pieces and puts them into pre-training model to extract answers. The existing recurrent chunking model suffers from unreasonable information extraction within segments and insufficient semantic transmission between segments. To solve these problems, a recurrent chunking model for long-text machine reading comprehension based on keywords is proposed. The knowledge base is introduced to embed the knowledge of articles and questions to enrich article word vector representntion in the word embedding stage. With the keywords, the article is divided into flexible segments via reinforcement learning strategies. The best answer of the article is finally obtained by balancing parameters. Experiments on CoQA, QuAC and TriviaQA demonstrate that the proposed model all achieves significant improvements according to F1 score compared with BERT-LARGE model and recurrent chunking model.
关键词
机器阅读理解 /
长文本 /
知识库 /
强化学习
{{custom_keyword}} /
Key words
machine reading comprehension /
long text /
knowledge base /
reinforcement learning
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all your need[C]//Proceedings of the 31st Conference on Neural Information Processing Systems, 2017: 5998-6008.
[2] DEVLIN J, CHANG M W, LEE K, et al.BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 4171-4186.
[3] HU M, F WEI, PENG Y, et al. Read+verify: Machine reading comprehension with unanswerable questions[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics,2017: 1436-1446.
[4] 陈峥,任建坤,袁浩瑞. 融合序列和图结构的机器阅读理解[J]. 中文信息学报,2021,35(04): 120-128.
[5] GONG H, SHEN Y, YU D, et al. Recurrent chunking mechanisms for long-text machine reading comprehension[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 6751-6761.
[6] GONG H, BHAT S, WU L, et al. Reinforcement learning based text style transfer without parallel training corpus[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, 2019: 3168-3180.
[7] ZEMLYANSKIY Y, AINSLIE J, DE JONG M, et al. Read Twice: Reading very large documents with memories[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,2021: 5189-5195.
[8] LIU X, LIU K, LI X, et al. An iterative multi-source mutual knowledge transfer framework for machine reading comprehension[C]//Proceedings of the 29th International Joint Conference on Artificial Intelligence,2020: 3794-3800.
[9] YANG A, WANG Q, LIU J, et al. Enhancing pre-trained language representations with rich knowledge for machine reading comprehension[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,2019: 2346-2357.
[10] GUAN J, HUANG F, ZHAO Z, et al. A knowledge-enhanced pretraining model for commonsense story generation[J]. Transaction of Associations for Computational Linguistics,2020,8(5): 93-108.
[11] MILLER, GEORGE A. WordNet: A lexical database for English[J]. Communications of ACM,1995, 38(11): 39-41.
[12] CARLSON A, BETTERIDGE J, KISIEL B, et al. Toward an architecture for never-ending language learning[C]//Proceedings of AAAI,2010: 1306-1313.
[13] REDDY S, CHEN D, MANNING C D. Coqa: A conversational question answering challenge[J]. Transactions of the Association for Computational Linguistics,2019,7: 249-266.
[14] CHOI E, HE H, IYYER M, et al. QuAC: Question answering in context [C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2018: 2174-2184.
[15] JOSHI M, CHOI E, WELD D, et al. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017: 1601-1611.
[16] IYER S, MIN S, MEHDAD Y, et al. RECONSIDER: Improvedre-ranking using span-focused cross-attention for open domain question answering[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,2021: 1280-1287.
[17] LEWIS P, PEREZ E, PIKTUS A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks[J]. Advances in Neural Information Processing Systems, 2020,33: 9459-9474.
[18] IZACARD G, GRAVE E. Leveraging passage retrieval with generative models for open domain question answering[C]//Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics,2021: 874-880.
[19] SACHAN D, PATWARY M, SHOEYBI M, et al. End-to-Endtraining of neural retrievers for open-domain question answering[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing,2021: 6648-6662.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
PDF(1500 KB)
