引入源端信息的机器译文自动评价方法研究

摘要
参考文献
相关文章 (15)

全文: PDF (1344 KB) HTML (1 KB)
输出: BibTeX | EndNote (RIS)

摘要机器译文自动评价是机器翻译中的一个重要任务。针对目前译文自动评价中完全忽略源语言句子信息，仅利用人工参考译文度量翻译质量的不足，该文提出了引入源语言句子信息的机器译文自动评价方法: 从机器译文与其源语言句子组成的二元组中提取描述翻译质量的质量向量，并将其与基于语境词向量的译文自动评价方法利用深度神经网络进行融合。在WMT-19译文自动评价任务数据集上的实验结果表明，该文所提出的方法能有效增强机器译文自动评价与人工评价的相关性。深入的实验分析进一步揭示了源语言句子信息在译文自动评价中发挥着重要作用。

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	罗琪
	李茂西

关键词 ：机器翻译, 译文自动评价, 质量向量, 语境词向量, 自然语言推断

Abstract：Automatic evaluation of machine translation is a key issue in machine translation. In existing work, the source sentence information is completely ignored and only the reference is used to measure the translation quality. This paper presents a novel automatic evaluation metric incorporating the source information: extracting the quality embeddings that describes the translation quality from a tuple consist of the machine translations and their corresponding source sentences, and incorporating it into the automatic evaluation method based on contextual embeddings by using a deep neural network. The experimental results on the dataset of WMT-19 Metrics task show that the proposed method can effectively enhance the evaluation correlation with the human judgments. Deep analysis further reveals that the information of the source sentences plays an important role in automatic evaluation of machine translation.

Key words： machine translation automatic evaluation of machine translation quality embeddings contextual embeddings natural language inference

收稿日期: 2021-02-21

中图分类号:

TP391

基金资助:国家自然科学基金(61662031, 61462044)；
江西省教育厅研究生创新基金（YC2020-S161）

作者简介: 罗琪（1996—），硕士研究生，主要研究领域为自然语言处理和机器翻译。E-mail: luoqi@jxnu.edu.cn; 李茂西（1977—），通信作者，博士，教授，主要研究领域为自然语言处理和机器翻译。E-mail: mosesli@jxnu.edu.cn

链接本文:

http://jcip.cipsc.org.cn/CN/Y2021/V35/I12/60

［1］Papineni Kishore, Roukos Salim, Ward Todd, et al. BLEU: a method for automatic evaluation of machine translation［C］//Proceedings of the ACL, 2002: 311-318.
［2］Doddington George. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics［C］//Proceedings of the HLT, 2002: 138-145.
［3］Popovi〖KG-*4〗c〖DD(-1*4〗〖HT6〗＇〖DD)〗 Maja, Ney Hermann. Syntax-oriented evaluation measures for machine translation output［C］//Proceedings of the WMT, 2009: 29-32.
［4］Banerjee Satanjeev, Lavie Alon. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments［C］//Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005: 65-72.
［5］Snover Matthew, Madnani Nitin, Dorr Bonnie, et al. TERp system description［C］//Proceedings of the Metrics MATR Workshop at AMTA, 2008: 104-108.
［6］Lo Chi-kiu . MEANT 2.0: accurate semantic MT evaluation for any output language［C］//Proceedings of the WMT, 2017: 589-597.
［7］Mikolov Tomas, Sutskever Ilya, Chen Kai, et al. Distributed representations of words and phrases and their compositionality［C］//Proceedings of the NIPS, 2013: 3111-3119.
［8］Boxing Chen, Hongyu Guo. Representation based translation evaluation metrics［C］//Proceedings of the ACL and IJCNLP. 2015: 150-155.
［9］Devlin Jacob, Chang Mingwei, Lee Kenton, et al. Bert: pre-training of deep bidirectional transformers for language understanding［C］//Proceedings of the NAACL, 2018: 4171-4186.
［10］Mathur Nitika, Baldwin Timothy, Cohn Trevor. Putting evaluation in context: contextual embeddings improve machine translation evaluation［C］//Proceedings of the ACL, 2019: 2799-2808.
［11］Gupta Rohit, Orasan Constantin, van Genabith Josef. Reval: a simple and effective machine translation evaluation metric based on recurrent neural networks［C］//Proceedings of the EMNLP, 2015: 1066-1072.
［12］Shimanaka Hiroki, Kajiwara Tomoyuki, Komachi Mamoru. Ruse: regressor using sentence embeddings for automatic machine translation evaluation［C］//Proceedings of the WMT, 2018: 751-758.
［13］Guzmán Francisco, Joty Shafiq, Màrquez Lluís, et al. Pairwise neural machine translation evaluation［C］//Proceedings of the ACL and IJCNLP, 2019: 805-814.
［14］Tai Kai Sheng, Socher Richard, Manning Christopher D. Improved semantic representations from tree-structured long short-term memory networks［C］//Proceedings of the ACL and IJCNLP, 2015: 1556-1566.
［15］Lili Men, Rui Men, Ge Li, et al. Natural language inference by tree-based convolution and heuristic matching［C］//Proceedings of the ACL, 2015: 130-136.
［16］Qian Chen, Xiaodan Zhu, Zhenhua Ling, et al. Enhanced ISTM for natural language inference［C］//Proceedings of the ACL, 2016: 1657-1668.
［17］Bahdanau Dzmitry, Cho Kyunghyun, Bengio Yoshua. Neural machine translation by jointly learning to align and translate［C］//Proceedings of the ICLR, 2014: 1-15.
［18］Kim Hyun, Jung Hunyoung, Kwon Hongseok, et al. Predictor-estimator: neural quality estimation based on target word prediction for machine translation［J］. ACM Transactions on Asian and Low-Resource Language Information Processing, 2017, 17(1): 1-22.
［19］Maoxi Li, Qingyu Xiang, Zhiming Chen, et al. A unified neural network for quality estimation of machine translation［J］. IEICE Transactons on Information and Systems, 2018, 101(9): 2417-2421.
［20］Vaswani Ashish, Shazeer Noam, Parmar Niki, et al. Attention is all you need［C］//Proceedings of the NIPS, 2017: 5998-6008.
［21］Kai Fan, Jiayi Wang, Bo Li, et al. “Bilingual expert” can find translation errors［C］//Proceedings of the AAAI, 2019,33: 6367-6374.
［22］Ziyang Wang, Hui Liu, Hexuan Chen, et al. NiuTrans submission for CCMT19 quality estimation task［C］//Proceedings of the CCMT, 2019: 82-92.
［23］Specia Lucia, Blain Frédéric, Logacheva Varvara, et al. Findings of the WMT 2018 shared task on quality estimation［C］//Proceedings of the WMT, 2018: 689-709.
［24］Qingsong Ma, Wei Johnny, Bojar 〖KG-*4〗[SX(B-*2/3]ˇ[]r[SX)]ej, et al. Results of the WMT19 metrics shared task: segment-level and strong MT systems pose big challenges［C］//Proceedings of the WMT, 2019: 62-90.
［25］Popovic Maja. chrF: character n-gram F-score for automatic MT evaluation［C］//Proceedings of the WMT, 2015: 392-395.
［26］Stanojevic Milo, Sima’an Khalil. Beer: Better evaluation as ranking［C］//Proceedings of the WMT, 2014: 414-419.
［27］Bojar Ond〖KG-*4〗[SX(B-*2/3]ˇ[]r[SX)]ej, Chatterjee Rajen, Federmann Christian, et al. Findings of the 2015 workshop on statistical machine translation［C］//Proceedings of the WMT, 2015: 1-46.
［28］Bojar Ond〖KG-*4〗[SX(B-*2/3]ˇ[]r[SX)]ej, Chatterjee Rajen, Federmann Christian, et al. Findings of the 2016 conference on machine translation［C］//Proceedings of the WMT, 2016: 131-198.
［29］Bojar Ond〖KG-*4〗[SX(B-*2/3]ˇ[]r[SX)]ej, Chatterjee Rajen, Federmann Christian, et al. Findings of the 2017 conference on machine translation［C］//Proceedings of the WMT, 2017: 169-214.