抽象语义表示是近年来国内外句子语义解析领域的研究热点,国际上已举办了CoNLL2019和CoNLL2020两届跨语言的评测。中文抽象语义表示评测是CoNLL2020的五大任务之一,取得了接近英语的解析效果,但是评测数据和评测指标仍有较大改进空间。为了推动中文抽象语义解析研究,该文在第二十一届中国计算语言学大会技术评测任务研讨会上组织了第二届评测,以新设计的Align-smatch指标为排名标准,采用改进的语义标注方案和标注语料库来进行评测。在基础测试集上,封闭模式的最高F1值为80.00%;盲测集上的表现则相比基础测试集下降了7个百分点左右。本次评测的最佳结果在MRP指标下比上届提高了2.66个百分点。统计发现,整体性能提升主要来源于概念之间的语义关系预测准确率的提高,而语义关系的对齐还有待提升。
Abstract
Meaning Representation Parsing EvaluationLI Bin1, XU Zhixing1, XIAO Liming2, ZHOU Junsheng3, QU Weiguang3, XUE Nianwen4
(1. School of Chinese Language and Literature, Nanjing Normal University, Nanjing, Jiangsu 210097, China;
2. Department of Chinese Language and Literature, Peking University, Beijing 100871, China;
3. School of Computer, Electronics and Information, Nanjing Normal
University, Nanjing, Jiangsu 210023, China;
4. Computer Science Department, Brandeis University, Waltham, MA 02453, US)
Abstract: Abstract Meaning Representation as a novel yet efficient approach of semantic parsing has prevailed the pertinent domains. English AMR corpus expanded as there has been two international conferences of cross-lingual semantic parsing evaluation held in CoNLL 2019 and CoNLL 2020 in the past several years. Chinese Abstract Meaning Representation parsing, as one of five tasks in CoNLL 2020, performed almost as well as in English. However, the data and metric used for evaluation can be improved. We hereby hold the Second Chinese Abstract Meaning Representation Parsing Evaluation to promote the research and further construct a better corpus for Chinese Abstract Meaning Representation. With regarding rules and preparations, we adopt high-quality data as the training set and use Align-smatch as the evaluation metric for the first time. According to F-score measurement, the best team scored a 0.8 in closed test and a 0.72 in open test under Align-smatch metric, respectively. And the top result surpasses the SOTA in CoNLL 2020 by 2.66 percentage points under MRP metric. Further study shows that this great progress mainly comes from better relation prediction between concepts, leaving the challenge of utilizing the alignments of semantic relations to be improved.
关键词
抽象语义表示 /
语义解析 /
评测指标 /
中文信息处理
{{custom_keyword}} /
Key words
abstract meaning representation /
semantic parsing /
evaluation metric /
Chinese information processing
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] 孙茂松,刘挺,姬东鸿,等.语言计算的重要国际前沿[J].中文信息学报, 2014, 28(1): 1-8.
[2] BANARESCU L, BONIAL C, CAI S, et al. Abstract meaning representation for sembanking[C]//Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, 2013: 178-186.
[3] OEPEN S, ABEND O, HAJIC J, et al. MRP: Cross-framework meaning representation parsing[C]//Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the Conference on Natural Language Learning, 2019: 1-27.
[4] OEPEN S, ABEND O, ABZIANIDZE L, et al. Proceedings of the CoNLL 2020 shared task: Cross-framework meaning representation parsing[C]//Proceedings of the CoNLL Shared Task: Cross-Framework Meaning Representation Parsing, 2020.
[5] ABZIANIDZE L, OEPEN S, ABEND O, et al. MRP: The 2nd shared task on crossframework and cross-lingual meaning representation parsing[C]//Proceedings of the CoNLL Shared Task: Cross-Framework Meaning Representation Parsing, 2020: 1-22.
[6] LI B, WEN Y, SONG L,et al. Building a Chinese AMR bank with concept and relation alignments[J]. Linguistic issues in language technology. 2019,18(1): 1-41.
[7] 戴玉玲,戴茹冰,冯敏萱,等.基于关系对齐的汉语虚词抽象语义表示与分析[J].中文信息学报, 2020, 34(04): 24-32.
[8] LI B, WEN Y, BU L, et al. Annotating the little prince with Chinese AMRs[C]//Proceedings of LAWX-the 10th Linguistic Annotation Workshop, 2016: 7-15.
[9] 李斌,闻媛,宋丽,等.融合概念对齐信息的中文AMR语料库的构建[J].中文信息学报,2017,31(06): 93-102.
[10] DAVID S, MILAN S. Permutation-invariant semantic parsing in perin[C]//Proceedings of the CoNLL Shared Task: Cross-Frame work Mearing Representation Parsing. 2020: 53-64.
[11] 肖力铭,李斌,许智星,等.基于概念关系对齐的中文抽象语义表示解析评测方法[J].中文信息学报,2022,36(01): 21-30.
[12] XIAO L, LI B, XU Z, et al. Align-smatch: A novel evaluation method for Chinese abstract meaning representation parsing based on alignment of concept and relation[C]//Proceedings of the 13th Language Resources and Evaluation Conference, 2022: 5938-5945.
[13] CAI S, KNIGHT K. Smatch: An evaluation metric for semantic feature structures[C]//Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 2013: 748-752.
[14] CUI Y, CHE W, LIU T, et al. Pre-training with whole word masking for Chinese Bert[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021(29): 3504-3514.
[15] HOANG T L, PICCO G, HOU Y, et al. Ensembling graph predictions for AMR parsing[C]//Proceedings of the 35th Conference on neural information processing systems, 2021(34): 8495-8505.
[16] CUI Y, CHE W, LIU T, et al. Revisiting pre-trained models for Chinese natural language processing[C]//Proceedings of the Association for Compntational Linguistics: EMNCP, 2020: 657-668.
[17] CUI Y, YANG Z, LIU T. PERT: Pretraining BERT with permuted language model[J]. arXiv preprint arXiv: 2203.06906, 2018.
[18] CHEN L, GAO B, CHANG B. A two-stage method for Chinese AMR parsing[C]//Proceedings of CAMRP,2022.
[19] KONSTAS I, IYER S, YATSKAR M, et al. Neural AMR: Sequence-to-sequence models for parsing and generation[C]//Proceedings of the 55th Annual Meething of the Assountion for Computational Linguistics, 2017: 146-157.
[20] VASWANI A,SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6000-6010.
[21] VINYALS O, FORTUNATO M, JAITLY N. Pointer networks[C]//Proceedings of the 28th International Conference Neural Information Processing Systems, 2015: 1-9.
[22] BEVILACQUA M,BLLOSHMI R, NAVIGLI R. One SPRING to rule them both: Symmetric AMR semantic parsing and generation without a complex pipeline[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021: 12564-12573.
[23] LEWIS M, LIU Y, GOYAL N, et al. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 7871-7880.
[24] ZHANG S, MA X, DUH K, et al. AMR parsing as sequence-to-graph transduction[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019: 80-94.
[25] DAMONTE M, COHEN S B, SATTA G. An incremental parser for abstract meaning representation[C]//Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 2017: 536-546.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
国家社会科学基金(18BYY127);国家自然科学基金(61772278)
{{custom_fund}}
PDF(1960 KB)
