动词驱动事件的共指关系中文语料库构建及大模型评测
Chinese Coreference Corpus from Verb-Driven Events and Its Application for LLM Evaluation
-
摘要: 事件共指关系研究是自然语言处理领域的一项基础任务。现有工作主要在封闭域上对事件共指进行研究。为了在开放域上研究事件共指关系,该文提出直接从事件的表达载体出发定义事件共指的研究思路,并具体讨论了动词驱动的事件之间的共指关系。该文构建了动词驱动事件共指关系的数据规范,并将共指关系细分为强共指与弱共指。基于数据规范,人工标注构建了38万余字的中文语料库,含1 000篇文档,29 250个动词,1 858条强共指链,994个弱共指动词对。并进行了流水线共指消解实验:提及检测阶段,采用成分句法分析算法提取动词成分,F1值为96.9%;共指关系预测阶段,分别采用提示工程、指令微调、知识蒸馏评测了大语言模型在该任务上的表现,比较模型预测的平均F1值,其中,提示工程方法为50.0%,指令微调方法为64.4%,知识蒸馏方法为71.7%。将经指令微调的大模型与经有监督微调的小模型进行对比,小模型的推理表现略低于大模型,推理效率远高于大模型。Abstract: The study of event coreference is a fundamental task in the field of natural language processing. To investigate coreference between all events in open domains, an approach that defines event coreference directly from the event expressions is proposed, with a specific focus on verb-driven events. A guideline for verb-driven event coreference is further composed, which divides event coreference into strong and weak coreference. Then, a manually annotated Chinese corpus consisting of over 380 thousand characters is constructed, comprising 1,000 documents, 29,250 verbs, 1,858 strong coreference chains and 994 weak coreference verb pairs. A pipeline experiments for coreference resolution is conducted, which demonstrate an F1 score of 96.9 in mention detection phase, and average F1 scores of 50.0, 64.4, and 71.7 for coreference prediction via prompt engineering, instruction tuning and knowledge distillation of LLM, respectively.
下载: