传统的文档级关系抽取方法在特征表示的有效性和噪声消除方面存在局限,不能准确地找出证据句子和实体对的关系。为了进一步提升文档级关系抽取和证据句子抽取的准确性,该文提出了一种使用小波变换对预训练语言模型生成的文本向量进行特征提取、清洗和去噪处理的方法。首先利用预训练语言模型对文档进行编码,将得到的初始文本向量应用小波变换出更精确的特征,其次引入多头注意力机制对小波变换的数据进行加权处理,以凸显与实体对关系相关的重要特征。为了充分利用原始数据和清洗后的数据,采用残差连接的方式将它们进行融合。在DocRED数据集上对模型进行了实验,结果表明,该文所提模型能够更好地抽取实体对的关系。
Abstract
Traditional methods of document-level relation extraction have limitations in the effectiveness of feature representation and noise elimination. To address this issue, this paper proposes a method that utilizes wavelet transform to extract, clean, and denoise text vectors generated by pre-trained language models. Firstly, the document is encoded by a pre-trained language model, and the obtained initial text vectors are applied to wavelet transform to obtain more precise features. Next, a multi-head attention mechanism is introduced to weight the data from wavelet transform, highlighting the important features relevant to entity relationships. To fully utilize both original and cleaned data, a residual connection is employed to fuse them together. Experiment on the DocRED dataset demonstrate that the proposed method performs better in extracting relationships between entity pairs.
关键词
文档级关系抽取 /
小波变换 /
多头注意力机制
{{custom_keyword}} /
Key words
document-level relationship extraction /
wavelet transform /
multi-head attention mechanism
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] 贾宝林,尹世群,王宁朝.基于门控多层感知机的端到端实体关系联合抽取[J].中文信息学报,2023,37(03): 143-151.
[2] ZENG D,LIU K,LAI S,et al.Relation classification via convolutional deep neural network[C]//Proceedings of the 25th International Conference on Computational Linguistics,2014: 2335-2344.
[3] XU W,CHEN K,ZHAO T.Document-level relation extraction with reconstruction[C]//Proceedings of the AAAI Conference on Artificial Intelligence,2021: 14167-14175.
[4] YU H,LI H,MAO D,et al.A relationship extractionmethod for domain knowledge graph construction[J].World Wide Web,2020,23: 735-753.
[5] YANG Z,WANG Y,GAN J,et al.Design and research of intelligent question-answering (Q & A) system based on high school course knowledge graph[J].Mobile Networks and Applications,2021: 1-7.
[6] DEVLIN J,ZHANG M,LEEK,et al.Bert: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,2019: 4171-4186.
[7] LIU Y,OTT M,GOYAL N,et al.RoBERTa: A robustly optimized bert pretraining approach[C]//Proceedings of the 20th Chinese National Conference on Computational Linguistics,2021: 1218-1227.
[8] VASWANI A,SHAZEER N,PARMARN,et al.Attention is all you need[C]//Proceedings of the 31st International Conference on in Neural Information Processing Systems,2017: 6000-6010.
[9] ZHANG Y,ZHONG V,CHEN D,et al.Position-aware attention and supervised data improve slot filling[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing,2017: 35-45.
[10] WOLF T,DEBUT L,SANH V,et al.Transformers: State-of-the-art natural language processing[C]//Proceedings of the conference on empirical Methods in Natural Language Processing: System Demonstrations,2020: 38-45.
[11] VERGA P,STRUBELL E,MCCALLUM A. Simultaneously self-attending to all mentions for full-abstract biological relation extraction[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,2018: 872-884.
[12] ZHOU W,HUANG K,MA T,et al.Document-level relation extraction with adaptive thresholding and localized context pooling[C]//Proceedings of the AAAI Conference on Artificial Intelligence,2021,35(16): 14612-14620.
[13] XU B,WANG Q,LYU Y,et al.Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction[C]//Proceedings of the AAAI Conference on Artificial Intelligence,2021,35(16): 14149-14157.
[14] XU W,CHEN K,MOU L,et al.Document-level relation extraction with sentences importance estimation and focusing[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,2022: 2920-2929.
[15] YU J,YANG D,TIAN S.Relation-specific attentions over entity mentions for enhanced document-level relation extraction[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,2022: 1523-1529.
[16] TAN Q,HE R,BING L,et al.Document-level relation extraction with adaptive focal loss and knowledge distillation[G]//Findings of the Association for Computational Linguistics,2022: 1672-1681.
[17] HUANG K,QI P,WANG G,et al.Entity and evidence guided document-level relation extraction[C]//Proceedings of the 6th Workshop on Representation Learning for NLP,2021: 307-315.
[18] XIE Y,SHEN J,LI S,et al.Eider: empowering document-level relation extraction with efficient evidenceextraction and inference-stage fusion[C]//Proceedings of the Association for Computational Linguistics,2022: 257-268.
[19] MA Y,WANG A,OKAZAKI N.DREEAM: guiding attention with evidence for improving document-level relation extraction[C]//Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,2023: 1971-1983.
[20] MAHAJAN A,JAT S,ROY S.Feature selection for short text classification using wavelet packettrans form[C]//Proceedings of the 19th Conference on Computational Natural Language Learning,2015: 321-326.
[21] YESILLI M C,KHASAWNEH F A,OTTO A.On transfer learning for chatter detection in turning using wavelet packet transform and ensemble empirical mode decomposition[J].CIRP Journal of Manufacturing Science and Technology,2020,28: 118-135.
[22] FRUSQUE G,FINK O.Robust time series denoising with learnable wavelet packet transform[J].arXiv preprint arXiv: 2206.06126,2022.
[23] FRUSQUE G,FINK O.Learnable wavelet packet transform for data-adapted spectrograms[C]//Proceedings of the International Conference on Acoustics,Speech and Signal Processing. IEEE,2022: 3119-3123.
[24] YAO Y,YE D,LI P,et al.DocRED: A large-scale document-level relation extraction dataset[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,2019: 764-777.
[25] RUDER S.An overview of multi-task learning in deepneural networks[J].arXiv preprint arXiv: 1706.05098,2017.
[26] Livio Baldini Soares, Nicholas Fitz Gerald, Jeffrey Ling, et al.Matching the Blanks: Distributional Similarity for Relation Learning[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,2019: 2895-2905.
[27] Robin Jia, Cliff Wong, Hoifung Poon. Document-Level Nary Relation Extraction with Multiscale Representation Learning[C]// Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics,2019: 3693-3704.
[28] ZHONG X,SUN A,CAMBRIA E.Time expression analysis and recognition using syntactic token types and general heuristic rules[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics,2017: 420-429.
[29] WOLPERT D H.Stacked generalization[J].Neural Networks,1992,5(2): 241-259.
[30] LOSHCHILOV I,HUTTER F.Decoupled weight decay regularization[J].arXiv preprint arXiv: 1711.05101,2017.
[31] GOYAL P,DOLLáR P,GIRSHICK R,et al.Accurate,large minibatch sgd: Training imagenet in 1 hour[J].arXiv preprint arXiv: 1706.02677, 2017.
[32] SOARES L B,FITZGERALD N,LING J,et al.Matching the blanks: Distributional similarity for relation learning[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics,2019: 2895-2905.
[33] PENG H,GAO T,HAN X,et al.Learning from context or names? An empirical study on neural relation extraction[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing,2020: 3661-3672.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
湖北省自然科学基金(E1KF291005);云南省自然科学基金(2022KZ00125)
{{custom_fund}}
PDF(1944 KB)
