研究发现对大规模的数据进行预训练可以更好地从自然语言文本中捕捉更丰富的语义信息,目前很多的多文档摘要的工作也应用了预训练模型并取得了一定的效果。但是这些预训练模型没有考虑到结构化的实体-关系信息。不能更好地从文本中捕获事实性知识。该文提出了基于实体信息增强和多粒度融合的多文档摘要模型MGNIE,该方法将实体关系信息融入预训练模型ERNIE中,增强知识事实以获得多层语义信息,解决摘要生成的事实一致性问题,进而从多种粒度进行多文档层次结构的融合建模,以词信息、实体信息以及句子信息捕捉长文本信息摘要生成所需的关键信息点。该文设计的模型在国际标准评测数据集MultiNews上的实验证明,所提模型对比强基线模型效果和竞争力获得较大提升。
Abstract
The pre-trained language model has been applied in multi-document summarization with positive results. Considering the defects of pre-trained models in capturing factual knowledge the document structure, this paper proposes a multi-document summarization model MGNIE based on entity information enhancement and multi-granularity fusion. It incorporates the entity relationship information into the pre-trained model ERNIE, enhancing knowledge facts to solve the factual consistency issue. In addition, the multi-document structure information is fused with multiple granularities in terms of words, entities and sentences. The proposed model achieves a significant improvement in effectiveness and competitiveness over the strong baseline model on the international standard evaluation dataset MultiNews.
关键词
实体信息增强 /
预训练语言模型 /
多粒度融合
{{custom_keyword}} /
Key words
entity information augmentation /
pre-trained language models /
multi-granularity fusion
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] LIU P J, SALEH M, POT E, et al. Generating wikipedia by summarizing long sequences[C]//Proceedings of ICLR, 2018: 1-18.
[2] FABBRI A R, LI I, SHE T, et al. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model[C]//Proceedings of ACL,2019: 1074-1084.
[3] DEVLIN J, Chang M W, LEE K, TOUTANOVA K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 4171-4186.
[4] LIU Y, LAPATA M. Hierarchical transformers for multi-document summarization[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019:5070-5081.
[5] ZHANG Z, HAN X, LIU Z, et al. ERNIE: Enhanced language representation with informative entities[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,2019:1441-1451.
[6] BORDES A, USUNIER N, GARCIA-DURAN A, et al. Translating embeddings for modeling multi-relational data[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems, 2013:2787-2795.
[7] ERKAN G, RADEV D R. Lexrank: Graph-based lexical centrality as salience in text summarization[J]. Journal of Artificial Intelligence Research, 2004, 22: 457-479.
[8] CHRISTENSEN J, SODERLAND S, ETZIONI O. Towards coherent multi-document summarization[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2013: 1163-1173.
[9] YASUNAGA M, ZHANG R, MEELU K, et al. Graph-based neural multi-document summarization[C]//Proceedings of the 21st Conference on Computational Natural Language Learning, 2017:452-462.
[10] ZHENG X, SUN A, LI J, et al. Subtopic-driven multi-document summarization[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019: 3153-3162.
[11] ALAMBO A, LOHSTROH C, MADAUS E, et al. Topic-centric unsupervised multi-document summarization of scientific and news articles[C]//Proceedings of the IEEE International Conference on Big Data. IEEE, 2020: 591-596.
[12] PASUNURU R, CELIKYILMAZ A, GALLEY M, et al. Data augmentation for abstractive query-focused multi-document summarization[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021: 13666-13674.
[13] LI W, XIAO X, LIU J, et al. Leveraging graph to improve abstractive multi-document summarization[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020:6232-6243.
[14] ZHANG X, WEI F, ZHOU M. HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019:5059-5069.
[15] DONG Y, WIETING J, VERGA P. Faithful to the document or to the world? Mitigating hallu-cinations via entity-linked knowledge in ab-stractive summarization[C] //Proceedings of the Association for Computational Linguistics: EMNLP,2022:1067-1082.
[16] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017:6000-6010.
[17] ZHAO J, LIU M, GAO L, et al. Summpip: Unsupervised multi-document summarization with sentence graph compression[C]//Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020: 1949-1952.
[18] JIN H, WANG T, WAN X. Multi-granularity interaction network for extractive and abstractive multi-document summarization[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 6244-6254.
[19] HENDRYCKS D, GIMPEL K. Gaussian error linear units (gelus)[J/OL], arXiv preprint arXiv: 1606.08415, 2016.
[20] CARBONELL J, GOLDSTEIN J. The use of MMR, diversity-based reranking for reordering documents and producing summaries[C]//Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 1998: 335-336.
[21] SEE A, LIU P J, MANNING C D. Get to the point: Summarization with pointer-generator net-works[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017:1073-1083.
[22] PEREZ BELTRACHINI L, LAPATA M. Multi-document summarization with determinantal point process attention[J]. Journal of Artificial Intelligence Research, 2021, 71: 371-399.
[23] KINGMA D P, BA J. Adam: A method for stochastic optimization[C]//Proceedings of the 3rd International Conference for Learning Representations, 2015:1-15.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
国家自然科学基金(61702091);黑龙江省自然科学基金(LH2022F002)
{{custom_fund}}
PDF(1180 KB)
