中圖分類號(hào)：TP391 文獻(xiàn)標(biāo)志碼：A DOI: 10.16157/j.issn.0258-7998.245391
中文引用格式： 韋浩翔，高盛祥，余正濤，等. 融合圖文預(yù)訓(xùn)練的漢越多模態(tài)神經(jīng)機(jī)器翻譯[J]. 電子技術(shù)應(yīng)用，2024，50(12)：48-54.
英文引用格式： Wei Haoxiang，Gao Shengxiang，Yu Zhengtao，et al. Chinese-Vietnamese multimodal neural machine translation with integrated image-text pre-training[J]. Application of Electronic Technique，2024，50(12)：48-54.

Chinese-Vietnamese multimodal neural machine translation with integrated image-text pre-training

Abstract： Due to significant grammatical differences and a scarcity of linguistic resources between Chinese and Vietnamese, the task of Chinese-Vietnamese neural machine translation faces challenges in the accurate translation of nouns. This paper proposes a novel multimodal neural machine translation method that integrates a text-based pre-trained model with a visual-linguistic joint pre-training model. The text-based model captures deep linguistic structures and semantics, while the visual-linguistic joint training model provides visual context related to the text, which helps the model understand and translate nouns more accurately. The two models are combined through a streamlined and efficient mapping network and dynamically integrate multimodal information via a Gumbel gating module to optimize translation outputs. In both Chinese-Vietnamese and Vietnamese-Chinese translation tasks, this method has achieved improvements of 7.13 and 4.27 BLEU points, respectively, compared to the traditional Transformer model.

Key words : Chinese-Vietnamese neural machine translation；vision-language joint pre-training；multimodal；attention