机器译文自动评价对推动机器翻译发展和应用有着重要作用。最新的神经机器译文自动评价方法使用预训练语境词向量提取深层语义特征,并将它们直接拼接输入多层神经网络预测译文质量,其中直接拼接操作容易导致特征间缺乏深入融合,而逐层抽象进行预测时容易丢失细粒度准确匹配信息。针对以上问题,该文提出将中期信息融合方法和后期信息融合方法引入译文自动评价,使用拥抱融合对不同特征进行交互中期融合,基于细粒度准确匹配的句移距离和句级余弦相似度进行后期融合。在WMT’21 Metrics Task基准数据集上的实验结果表明,提出的方法能有效提高其与人工评价的相关性,达到与参加评测最优系统的可比性能。
Abstract
Machine translation evaluation plays an important role in promoting the development and application of machine translation. The latest neural methods of evaluating machine translation use pretrained contextual embeddings to extract different deep semantic features, and then simply concatenate them feed into the multi-layer neural network to predict translation quality. We propose to introduce middle stage information fusion and late stage information fusion into evaluation of machine translation. More specifically, we propose to use embrace fusion to interactively fuse different features in the middle stage. In the late stage, we fuse sentence mover’s distance and sentence cosine similarity based on fine-grained accurate matching. Experimental results on the WMT'21 Metrics Task show that the proposed method can achieve competitive performance with the best metrics in the evaluation campaign.
关键词
机器翻译 /
译文自动评价 /
信息融合 /
信息表征 /
拥抱融合
{{custom_keyword}} /
Key words
machine translation /
automatic evaluation of machine translation /
information fusion /
information representation /
embrace fusion
{{custom_keyword}} /
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] SHIWEN Y. Automatic evaluation of output quality for machine translation systems[J]. Machine Translation, 1993, 8(1): 117-126.
[2] PAPINENI K, ROUKOS S, WARD T, et al. BLEU: A method for automatic evaluation of machine translation[C]//Proceedings of the ACL, 2002: 311-318.
[3] 张家俊,赵阳,宗成庆(译).神经机器翻译[M].北京:机械工业出版社,2022.
[4] 刘洋, 刘群, 林守勋. 机器翻译评测中的模糊匹配[J]. 中文信息学报, 2005, 19(3): 46-54.
[5] JOSEPH P.TURIAN, LUKE S, et al. Evaluation of machine translation and its evaluation[C]//Proceedings of the MT Summit, 2003.
[6] LIN C Y. Rouge: A package for automatic evaluation of summaries[G]//Proceedings of the Text Summarization Branches Out, 2004: 74-81.
[7] SU K Y, WU M W, CHANG J S. A new quantitative quality measure for machine translation systems[C]//Proceedings of the COLING, 1992: 433-439.
[8] SNOVER M, DORR B, SCHWARTZ R, et al. A study of translation edit rate with targeted human annotation[C]//Proceedings of the AMTA, 2006: 223-231.
[9] CHEN B, GUO H. Representation based translation evaluation metrics[C]//Proceedings of the ACL and IJCNLP, 2015: 150-155.
[10] KUSNER M, SUN Y, KOLKIN N, et al. From word embeddings to document distances[C]//Proceedings of the ICML, 2015: 957-966.
[11] LO C.YiSi-a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources[C]//Proceedings of the WMT, 2019: 507-513.
[12] ZHANG T, KISHORE V, WU F, et al. Bertscore: Evaluating text generation with bert[C]//Proceedings of ICLR 2020.2020:1-43.
[13] SHIMANAKA H, KAJIWARA T, KOMACHI M. Ruse: Regressor using sentence embeddings for automatic machine translation evaluation[C]//Proceedings of the WMT, 2018: 751-758.
[14] SELLAM T, DAS D, PARIKH A P. BLEURT: Learning robust metrics for text generation[J/OL]. arXiv preprint arXiv:2004.04696, 2020.
[15] CHOI J H, LEE J S. EmbraceNet: A robust deep learning architecture for multimodal classification[J]. Information Fusion, 2019, 51: 259-270.
[16] SONG Y, ZHAO J, SPECIA L. SentSim: Crosslingual semantic evaluation of machine translation[C]//Proceedings of the NAACL, 2021: 3143-3156.
[17] ZHAO W,PEYRARD M, LIU F, et al. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance[C]//Proceedings of the EMNLP and IJCNLP, 2019: 563-578.
[18] WAN Y, LIU D, YANG B, et al.RoBLEURT submission for the WMT2021 metrics task[C]//Proceedings of the WMT, 2021: 1053-1058.
[19] MATHUR N, BALDWIN T, COHN T. Putting evaluation in context: Contextual embeddings improve machine translation evaluation[C]//Proceedings of the ACL, 2019: 2799-2808.
[20] LUO Q, LI M. Research on incorporating the source information to automatic evaluation of machine translation[C]//Proceedings of the CCL, 2020: 414-423.
[21] HU W, LI M,QIU B, et al. Neural automatic evaluation of machine translation method combined with XLM word representation[C]//Proceedings of the CCL, 2021: 13-22.
[22] LI M X, XIANG Q Y, CHEN Z M, et al. A unified neural network for quality estimation of machine translation[J]. IEICE Transactions on Information and Systems, 2018, 101(9): 2417-2421.
[23] LAMPLE G, CONNEAU A. Cross-lingual language model pretraining[C]//Proceedings of the NeurIPS, 2019: 7059-7069.
[24] MOU L, MEN R, LI G, et al. Natural language inference by tree-based convolution and heuristic matching[C] //Proceedings of the ACL, 2016: 130-136.
[25] REIMERS N,GUREVYCH I. Sentence-bert: Sentence embeddings using Siamese bert-networks[C]//Proceedings of the EMNLP and IJCNLP, 2019: 3982-3992.
[26] FREITAG M, REI R, MATHUR N, et al. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain[C]//Proceedings of the WMT, 2021: 733-774.
[27] 翟煜锦, 李培芸, 项青宇, 等. 基于 QE 的机器翻译重排序方法研究[J]. 江西师范大学学报(自然科学版), 2020, 44(1): 46-50.
[28] POPOVIC＇ M. chrF: Character n-gram F-score for automatic MT evaluation[C]//Proceedings of the SMT, 2015: 392-395.
[29] THOMPSON B, POST M. Automatic machine translation evaluation in many languages via zero-shot paraphrasing[C]//Proceedings of the EMNLP, 2020: 90-121.
[30] 赵阳, 周龙, 王迁, 等. 民汉稀缺资源神经机器翻译技术研究[J]. 江西师范大学学报 (自然科学版), 2019, 43(6): 630-637.
{{custom_fnGroup.title_cn}}
脚注
{{custom_fn.content}}
基金
国家自然科学基金(61662031,61462044);江西省教育厅科技项目(GJJ210306);教育部产学合作协同育人项目(220604647062739)
{{custom_fund}}
PDF(2720 KB)
