|
|
|
| A Novel Evaluation Method for Chinese Abstract Meaning Representation Parsing Based on Alignment of Concept and Relation |
XIAO Liming1 , LI Bin1 , XU Zhixing1 , HUO Kairui1 ,
FENG Minxuan1 , ZHOU Junsheng2 , QU Weiguang2 |
1.School of Chinese Language and Literature, Nanjing Normal University, Nanjing, Jiangsu 210097, China;
2.School of Computer, Electronics and Information, Nanjing Normal University, Nanjing, Jiangsu 210023, China |
|
|
|
|
Abstract Abstract Meaning Representation is a sentence-level meaning representation, which abstracts a sentence’s meaning into a rooted acyclic directed graph. With the continuous expansion of Chinese AMR corpus, more and more scholars have developed parsing systems to automatically parse sentences into Chinese AMR. To make up for the vacancy of Chinese AMR parsing evaluation methods, we have improved the Smatch algorithm of generating triples to make it compatible with concept alignment and relation alignment. We finally complete a new integrity metric Align-Smatch for paring evaluation. Compared on 100 manually annotated AMR and gold AMR, it is revealed that Align-Smatch works well in alignments and more robust in evaluating arcs. We also put forward some fine-grained metric for evaluating concept alignment, relation alignment and implicit concepts, in order to further measure parsers’ performance in subtasks.
|
|
Received: 30 March 2021
|
|
|
|
|
|
[1] 孙茂松, 刘挺, 姬东鸿, 等.语言计算的重要国际前沿[J].中文信息学报, 2014, 28(1): 1-8.
[2] Banarescu L, Bonial C, Cai S, et al. Abstract meaning representation for sembanking[C]//Proceedings of the Linguistic Annotation Workshop and Interoperability with Discourse, 2013: 178-186.
[3] 戴玉玲, 戴茹冰, 冯敏萱, 等.基于关系对齐的汉语虚词抽象语义表示与分析[J].中文信息学报, 2020, 34(04): 24-32.
[4] Flanigan J, Thomson S,Carbonell J, et al. A discriminative graph-based parser for the abstract meaning representation[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, 2014: 1426-1436.
[5] Liu Y, Che W, Zheng B, et al. An AMR aligner tuned by transition-based parser[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2018: 2422-2430.
[6] Lyu C, Titov I. AMR parsing as graph prediction with latent alignment[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018: 397-407.
[7] Zhang S, Ma X, Duh K, et al. AMR parsing as sequence-to-graph transduction[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019: 80-94.
[8] Li B, Wen Y, Bu L, et al. Annotating the Little Prince with Chinese AMRs[C]//Proceedings of the Linguistic Annotation Workshop, 2016: 7-15.
[9] 李斌, 闻媛, 宋丽,等.融合概念对齐信息的中文AMR语料库的构建[J].中文信息学报, 2017, 31(6): 97-106.
[10] Wang C, Li B, Xue N. Transition-based Chinese AMR parsing[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018: 247-252.
[11] 顾敏.基于转移神经网络的中文AMR解析研究[D].南京: 南京师范大学硕士学位论文, 2018.
[12] 吴泰中, 顾敏, 周俊生, 等.基于转移神经网络的中文AMR解析[J].中文信息学报, 2019, 33(4): 1-11.
[13] Damonte M, Cohen S B. Cross-lingual abstract meaning representation parsing[C]//Proceedings of the
Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018: 1146-1155.
[14] Blloshmi R, Tripodi R, Navigli R. XL-AMR: Enabling cross-lingual AMR parsing with transfer learning techniques[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2020: 2487-2500.
[15] Oepen S, Abend O, Abzianidze L, et al. MRP 2020: The second shared task on cross-framework and cross-lingual meaning representation parsing [C]//Proceedings of the CoNLL Shared Task: Cross-Framework Meaning Representation Parsing, 2020: 1-22.
[16] Samuel D, Straka M. FAL at MRP 2020: Permutation-invariant semantic parsing in PERIN[C]// Proceedings of the CoNLL Shared Task: Cross-Framework Meaning Representation Parsing, 2020: 53-64.
[17] Cai S, Knight K. Smatch: An evaluation metric for semantic feature structures[C]//Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 2013: 748-752.
[18] Song L, Gildea D.SemBleu: A robust metric for AMR parsing evaluation[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019: 4547-4552.
[19] Damonte M, Cohen S B, Satta G. An incremental parser for abstract meaning representation[C]//Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 2017: 536-546.
[20] Cai D, Lam W. Core semantic first: A top-down approach for AMR parsing[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019: 3790-3800.
[21] Pourdamghani N, Gao Y, Hermjakob U, et al. Aligning English strings with abstract meaning representation graphs[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2014: 425-429.
[22] Brandt L, Grimm D, Zhou M, et al.Icl-hd at semeval-2016 task 8: Meaning representation parsing-augmenting AMR parsing with a preposition semantic role labeling neural network[C]//Proceedings of the 10th International Workshop on Semantic Evaluation, 2016: 1160-1166.[23] Barzdins G, Gosko D. Riga at semeval-2016 task 8: Impact of smatch extensions and character-level neural translation on AMR parsing accuracy[C]// Proceedings of the 10th International Workshop on Semantic Evaluation, 2016: 1143-1147.
[24] Konstas I, Iyer S, Yatskar M, et al. Neural AMR: Sequence-to-sequence models for parsing and generation[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017: 146-157.
[25] abokrtsk Z, Zeman D, evíková M. Sentence meaning representations across languages: What can we learn from existing frameworks?[J].Computational Linguistics, 2020, 46(1): 1-61. |
|
|
|
2022, Vol. 36 
