[1] GEHRING J, AULI M, GRANGIER D, et al. Convolutional sequence to sequence learning[C]//Proceedings of the 34th International Conference on Machine Learning, 2017: 1243-1252.
[2] JORDAN M I. Serial order: A parallel distributed processing approach[M]. Advances in Psychology, North-Holland, 1997, 121: 471-495.
[3] VASWANI A, SHAZEER N M, PARMAR N, et al. Attention is all you need[C]//Proceedings of the Advances in Neural Information Processing Systems, 2017: 6000-6010.
[4] 肖桐,朱靖波.机器翻译: 基础与模型[M].北京: 电子工业出版社, 2021: 373-374.
[5] KENTON J D M W C, TOUTANOVA L K. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of NAACL-HLT, 2019: 4171-4186.
[6] CONNEAU A, LAMPLE G. Cross-lingual language model pretraining[C]//Proceedings of the Advances in Neural Information Processing Systems, 2019.
[7] RADFORD A, WU J, CHILD R, et al. Language models are unsupervised multitask learners[J]. OpenAI Blog, 2019, 1(8): 9-17.
[8] SONG K, TAN X, QIN T, et al. Mass: Masked sequence to sequence pre-training for language generation[J]. arXiv preprint arXiv: 1905.02450, 2019.
[9] LEWIS M, LIU Y, GOYAL N, et al. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation and comprehension[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 7871-7880.
[10] LI J, TANG T, ZHAO W X, et al. Pretrained language models for text generation: A survey[C]//Proceedings of the 13th International Joint Conference on Artificial Intelligence, 2021.
[11] QIU X, SUN T, XU Y, et al. Pre-trained models for natural language processing: A survey[J]. Science China Technological Sciences, 2020, 63(10): 1872-1897.
[12] YUAN S, ZHAO H, ZHAO S, et al. A roadmap for big model[J]. arXiv preprint arXiv: 2203.14101, 2022.
[13] WANG M, LI L. Pre-training methods for neural machine translation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Tutorial Abstracts, 2021: 21-25.
[14] Liu X, Wang L, Wong D F, et al. On the copying behaviors of pre-training for neural machine translation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2021: 4265-4275.
[15] REN S, WU Y, LIU S, et al. Explicit cross-lingual pre-training for unsupervised machine translation[J]. arXiv preprint arXiv: 1909.00180, 2019.
[16] MAO Z, CHU C, KUROHASHI S. Linguistically driven multi-task pre-training for low-resource neural machine translation[J]. Transactions on Asian and Low-Resource Language Information Processing, 2022, 21(4): 1-29.
[17] RAFFEL C, SHAZEER N, ROBERTS A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer[J]. Journal of Machine Learning Research, 2020, 21(140): 1-67.
[18] LUO F, WANG W, LIU J, et al. VECO: Variable and flexible cross-lingual pre-training for language understanding and generation[J]. arXiv preprint arXiv: 2010.16046, 2020.
[19] LIU Y, GU J, GOYAL N, et al. Multilingual denoising pre-training for neural machine translation[J]. Transactions of the Association for Computational Linguistics, 2020, 8: 726-742.
[20] DABRE R, SHROTRIYA H, KUNCHUKUTTAN A, et al. Indicbart: A pre-trained model for natural language generation of indic languages[J]. arXiv preprint arXiv: 2109.02903, 2021.
[21] YANG J, MA S, ZHANG D, et al. Alternating language modeling for cross-lingual pre-training[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence, 2020: 9386-9393.
[22] ZHANG W, LI X, YANG Y, et al. Pre-training on mixed data for low-resource neural machine translation[J]. Information, 2021, 12(3): 133-139.
[23] YANG Z, HU B, HAN A, et al. Csp: Code-switching pre-training for neural machine translation[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2020: 2624-2636.
[24] LIN Z, PAN X, WANG M, et al. Pre-training multilingual neural machine translation by leveraging alignment information[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2020: 2649-2663.
[25] PAN X, WANG M, WU L, et al. Contrastive learning for many-to-many multilingual neural machine translation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 244-258.
[26] LI P, LI L, ZHANG M, et al. Universal conditional masked language pre-training for neural machine translation[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022: 6379-6391.
[27] FAN A, BHOSALE S, SCHWENK H, et al. Beyond English-centric multilingual machine translation[J]. Journal of Machine Learning Research, 2021, 22(107): 1-48.
[28] WANG S, TU Z, TAN Z, et al. Language models are good translators[J]. arXiv preprint arXiv: 2106,13627, 2021.
[29] REN S, ZHOU L, LIU S, et al. SemFace: Pre-training encoder and decoder with a semantic interface for neural machine translation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 4518-4527.
[30] LIN Z, WU L, WANG M, et al. Learning language specific sub-network for multilingual machine translation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 293-305.
[31] IMAMURA K, SUMITA E. Recycling a pre-trained BERT encoder for neural machine translation[C]//Proceedings of the 3rd Workshop on Neural Generation and Translation, 2019: 23-31.
[32] CLINCHANT S, JUNG K W, NIKOULINA V. On the use of BERT for neural machine translation[C]//Proceedings of the 3rd Workshop on Neural Generation and Translation, 2019: 108-117.
[33] MA S, YANG J, HUANG H, et al. XLM-T: Scaling up multilingual machine translation with pretrained cross-lingual transformer encoders[J]. arXiv preprint arXiv: 2012.15547, 2020.
[34] ROTHE S, NARAYAN S, SEVERYN A. Leveraging pre-trained checkpoints for sequence generation tasks[J]. Transactions of the Association for Computational Linguistics, 2020, 8: 264-280.
[35] MA S, DONG L, HUANG S, et al. DeltaLM: Encoder-decoder pre-training for language generation and translation by augmenting pretrained multilingual encoders[J]. arXiv preprint arXiv: 2106.13736, 2021.
[36] TANG Y, TRAN C, LI X, et al. Multilingual translation with extensible multilingual pretraining and finetuning[J]. arXiv preprint arXiv: 2008.00401, 2020.
[37] LIU Z, WINATA G I, FUNG P. Continual mixed-language pre-training for extremely low-resource neural machine translation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 2706-2718.
[38] GUO J, ZHANG Z, XU L, et al. Incorporating BERT into parallel sequence decoding with adapters[C]//Proceedings of the Advances in Neural Information Processing Systems, 2020: 10843-10854.
[39] SUN Z, WANG M, Li L. Multilingual translation via grafting pre-trained language models[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021: 2735-2747.
[40] STN A, BRARD A, BESACIER L, et al. Multilingual unsupervised neural machine translation with denoising adapters[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021: 6650-6662.
[41] STICKLAND A C, LI X, GHAZVININEJAD M. Recipes for adapting pre-trained monolingual and multilingual models to machine translation[C]//Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, 2021: 3440-3453.
[42] WENG R, YU H, LUO W, et al. Deep fusing pre-trained models into neural machine translation[C]//Proceedings of the 36th AAAI Conference on Artificial Intelligence, 2022.
[43] ZHU J, XIA Y, WU L, et al. Incorporating BERT into neural machine translation[J]. arXiv preprint arXiv: 2002.06823, 2020.
[44] PARK J, ZHAO H. Enhancing language generation with effective checkpoints of pre-trained language model[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 2686-2694.
[45] SHAVARANI H S, SARKAR A. Better neural machine translation by extracting linguistic information from BERT[C]//Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, 2021: 2772-2783.
[46] XU H, VAN D B, MURRAY K. BERT, mBERT, or BiBERT?: A study on contextualized embeddings for neural machine translation[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021: 6663-6675.
[47] YANG J, WANG M, ZHOU H, et al. Towards making the most of BERT in neural machine translation[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence, 2020: 9378-9385.
[48] WENG R, YU H, HUANG S, et al. Acquiring knowledge from pre-trained model to neural machine translation[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence, 2020: 9266-9273.
[49] TAN Z, ZHANG X, WANG S, et al. MSP: Multi-stage prompting for making pre-trained language models better translators[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022: 6131-6142.
[50] WEI X, WENG R, HU Y, et al. On learning universal representations across languages[J]. arXiv preprint arXiv: 2007.15960, 2020.
[51] ZHAO M, WU H, NIU D, et al. Reinforced curriculum learning on pre-trained neural machine translation models[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence, 2020: 9652-9659.
[52] TRAN C, TANG Y, LI X, et al. Cross-lingual retrieval for iterative self-supervised training[C]//Proceedings of the Advances in Neural Information Processing Systems, 2020: 2207-2219.
[53] WANG W, JIAO W, HAO Y, et al. Understanding and improving sequence-to-sequence pretraining for neural machine translation[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022: 2591-2600.
[54] SUSANTO R H, WANG D, YADAV S, et al. Rakutens participation in WAT: Examining the effectiveness of pre-trained models for multilingual and multimodal machine translation[C]//Proceedings of the 8th Workshop on Asian Translation, 2021: 96-105.
[55] VZQUEZ R, CELIKKANAT H, CREUTZ M, et al. On the differences between BERT and MT encoder spaces and how to address them in translation tasks[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 337-347.
[56] ZAN C, DING L, SHEN L, et al. On the complementarity between pre-training and random-initialization for resource-rich machine translation[C]//Proceedings of the 29th International Conference on Computational Linguistics, 2022: 5029-5034.
[57] HAN L, EROFEEV G, SOROKINA I, et al. Examining large pre-trained language models for machine translation: What you dont know about it[J]. arXiv preprint arXiv: 2209.07417, 2022.
[58] SONG H, DABRE R, MAO Z, et al. Pre-training via leveraging assisting languages and data selection for neural machine translation[J]. arXiv preprint arXiv: 2001.08353, 2020.
[59] RICHARDSON A, WILES J. A systematic study reveals unexpected interactions in pre-trained neural machine translation[C]//Proceedings of the 13th Language Resources and Evaluation Conference, 2022: 1437-1443.
[60] HUANG D, WANG K, ZHANG Y. A comparison between pre-training and large-scale back-translation for neural machine translation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 1718-1732.
[61] LIU X, WANG L, WONG D F, et al. On the complementarity between pre-training and back-translation for neural machine translation[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021: 2900-2907.
[62] SCHNEIDER S, BAEVSKI A, COLLOBERT R, et al. wav2vec: Unsupervised pre-training for speech recognition[C]//Proceedings of the Interspeech, 2019: 3465-3469.
[63] BAEVSKI A, ZHOU Y, MOHAMED A, et al. wav2vec 2.0: A framework for self-supervised learning of speech representations[C]//Proceedings of the Advances in Neural Information Processing Systems, 2020: 12449-12460.
[64] HSU W N, BOLTE B, TSAI Y H H, et al. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3451-3460.
[65] CHEN S, WANG C, CHEN Z, et al. WavLM: Large-scale self-supervised pre-training for full stack speech processing[J]. IEEE Journal of Selected Topics in Signal Processing, 2022, 16(6): 1505-1518.
[66] CHUNG Y A, ZHANG Y, HAN W, et al. W2v-BERT: Combining contrastive learning and masked language modeling for self-supervised speech pre-training[C]//Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop, 2021: 244-250.
[67] JI B, ZHANG Z, DUAN X, et al. Cross-lingual pre-training based transfer for zero-shot neural machine translation[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence, 2020: 115-122.
[68] LAKEW S M, NEGRI M, TURCHI M. Zero-shot neural machine translation with self-learning cycle[C]//Proceedings of the 4th Workshop on Technologies for MT of Low Resource Languages, 2021: 96-113.
[69] CHEN G, MA S, CHEN Y, et al. Zero-shot cross-lingual transfer of neural machine translation with multilingual pretrained encoders[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021: 15-26.
[70] CHEN G, MA S, CHEN Y, et al. Towards making the most of cross-lingual transfer for zero-shot neural machine translation[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022: 142-157.
[71] GUO Z, LE NGUYEN M. Document-level neural machine translation using BERT as context encoder[C]//Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop, 2020: 101-107.
[72] DONATO D, YU L, DYER C. Diverse pretrained context encodings improve document translation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 1299-1311.
[73] YANG P, ZHANG P, CHEN B, et al. Context-interactive pre-training for document machine translation[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, 2021: 3589-3595.
[74] CHEN L, LI J, GONG Z, et al. Improving context-aware neural machine translation with source-side monolingual documents[C]//Proceedings of the 13th International Joint Conference on Artificial Intelligence, 2021: 3794-3800.
[75] HU J, HAYASHI H, CHO K, et al. DEEP: DEnoising entity pre-training for neural machine translation[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022: 1753-1766.
[76] MARIE B, FUJITA A. Synthesizing monolingual data for neural machine translation[J]. arXiv preprint arXiv: 2101.12462, 2021.
[77] HAN J M, BABUSCHKIN I, EDWARDS H, et al. Unsupervised neural machine translation with generative language models only[J]. arXiv preprint arXiv: 2110.05448, 2021.
[78] VAN DER WERFF T, VAN NOORD R, TORAL A. Automatic discrimination of human and neural machine translation: A study with multiple pre-trained models and longer context[C]//Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, 2022: 161-170.
[79] GUO J, ZHANG Z, XU L, et al. Adaptive adapters: An efficient way to incorporate BERT into neural machine translation[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 1740-1751.
[80] BANSAL S, KAMPER H, LIVESCU K, et al. Pre-training on high-resource speech recognition improves low-resource speech-to-text translation[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologiess, 2019: 58-68.
[81] BéRARD A, BESACIER L, KOCABIYIKOGLU A C, et al. End-to-end automatic speech translation of audiobooks[C]//Proceedings of the International Conference on Acoustics, Speech and Signal Processing, 2018: 6224-6228.
[82] LIU Y, ZHANG J, XIONG H, et al. Synchronous speech recognition and speech-to-text translation with interactive decoding[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(05): 8417-8424.
[83] WANG C, WU Y, LIU S, et al. Curriculum pre-training for end-to-end speech translation[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 3728-3738.
[84] ALINEJAD A, SARKAR A. Effectively pretraining a speech translation decoder with machine translation data[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2020: 8014-8020.
[85] DONG Q, WANG M, ZHOU H, et al. Consecutive decoding for speech-to-text translation[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(14): 12738-12748.
[86] LIU Y, XIONG H, HE Z, et al. End-to-end speech translation with knowledge distillation[J]. arXiv preprint arXiv: 1904.08075, 2019.
[87] STOIAN M C, BANSAL S, GOLDWATER S. Analyzing ASR pretraining for low-resource speech-to-text translation[C]//Proceedings of the ICASSP IEEE International Conference on Acoustics, Speech and Signal Processing, 2020: 7909-7913.
[88] WANG C, WU Y, LIU S, et al. Bridging the gap between pre-training and fine-tuning for end-to-end speech translation[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(05): 9161-9168.
[89] WANG C, WU A, PINO J, et al. Large-scale self-and semi-supervised learning for speech translation[J]. arXiv preprint arXiv: 2104.06678, 2021.
[90] WU A, WANG C, PINO J, et al. Self-supervised representations improve end-to-end speech translation[C]//Proceedings of the Interspeech, 2020: 1491-1495.
[91] CHEN J, MA M, ZHENG R, et al. Mam: Masked acoustic modeling for end-to-end speech-to-text translation[J]. arXiv preprint arXiv: 2010.11445, 2020.
[92] NGUYEN H, BOUGARES F, TOMASHENKO N, et al. Investigating self-supervised pre-training for end-to-end speech translation[C]//Proceedings of the Interspeech, 2020.
[93] CHUNG Y A, GLASS J. Generative pre-training for speech with autoregressive predictive coding[C]//Proceedings of the ICASSP IEEE International Conference on Acoustics, Speech and Signal Processing, 2020: 3497-3501.
[94] LE H, PINO J, WANG C, et al. Lightweight adapter tuning for multilingual speech translation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 817-824.
[95] DONG Q, YE R, WANG M, et al. Listen, understand and translate: Triple supervision decouples end-to-end speech-to-text translation[C]//Proceedings of the 35th AAAI Conference on Artificial Intelligence, 2021, 35(14): 12749-12759.
[96] ZHENG R, CHEN J, MA M, et al. Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation[C]//Proceedings of the 38th International Conference on Machine Learning, 2021: 12736-12746.
[97] YE R, WANG M, LI L. End-to-end speech translation via cross-modal progressive training[J]. arXiv preprint arXiv: 2104.10380, 2021.
[98] XU C, HU B, LI Y, et al. Stacked acoustic-and-textual encoding: Integrating the pre-trained models into speech translation encoders[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 2619-2630.
[99] BAPNA A, CHUNG Y, WU N, et al. SLAM: A unified encoder for speech and language modeling via speech-text joint pre-training[J]. arXiv preprint arXiv: 2110.10329, 2021.
[100] AO J, WANG R, ZHOU L, et al. SpeechT5: Unified-modal encoder-decoder pre-training for spoken language processing[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022: 5723-5738.
[101] LI X, WANG C, TANG Y, et al. Multilingual speech translation from efficient finetuning of pretrained models[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 827-838.
[102] ZHAI X, KOLESNIKOV A, HOULSBY N, et al. Scaling vision transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 12104-12113.
[103] RIQUELME C, PUIGCERVER J, MUSTAFA B, et al. Scaling vision with sparse mixture of experts[J]. Advances in Neural Information Processing Systems, 2021, 34: 8583-8595.
[104] LIU Z, HU H, LIN Y, et al. Swin transformer v2: Scaling up capacity and resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 12009-12019.
[105] CAGLAYAN O, KUYU M, AMAC M S, et al. Cross-lingual visual pre-training for multimodal machine translation[C]//Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, 2021: 1317-1324.
[106] YAWEI K, FAN K. Probing multi-modal machine translation with pre-trained language model[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021: 3689-3699.
[107] SONG Y, CHEN S, JIN Q, et al. Product-oriented machine translation with cross-modal cross-lingual pre-training[C]//Proceedings of the 29th ACM International Conference on Multimedia, 2021: 2843-2852.
[108] HIRASAWA T, KANEKO M, IMANKULOVA A, et al. Pre-trained word embedding and language model improve multimodal machine translation: A case study in multi30K[J]. IEEE Access, 2022, 10: 67653-67668.
[109] BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners[J]. Advances in Neural Information Processing Systems, 2020, 33: 1877-1901.