Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userBERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
بيرت، mbert، أو bibert؟دراسة حول تضمينات السياق للترجمة الآلية العصبية
318
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
دفع نجاح ترميزات ثنائية الاتجاه باستخدام نماذج لغة ملثم، مثل بيرت، في العديد من مهام معالجة اللغة الطبيعية، بباحثي المحاولة لإدماج هذه النماذج المدربة مسبقا في أنظمة الترجمة الآلية العصبية (NMT). ومع ذلك، فإن الأساليب المقترحة لإدماج النماذج المدربة مسبقا هي غير تافهة وتركز بشكل أساسي على بيرتف، والتي تفتقر إلى مقارنة التأثير الذي قد يكون له النماذج الأخرى المدربة مسبقا على أداء الترجمة. في هذه الورقة، نوضح ببساطة باستخدام الناتج (Attentralized Advedings) من طراز لغة تدرب مسبقا مخصص ومناسب (Bibert) مناسبة (Bibert) حيث أن إدخال ترميز NMT يحقق أداء ترجمة حديثة من بين الفن. علاوة على ذلك، نقترح أيضا نهج اختيار طبقة مؤشر استوكاستك ومفهوم نموذج الترجمة المزدوج الاتجاه لضمان الاستخدام الكافي للمشروعات السياقية. في حالة عدم استخدام الترجمة الخلفية، تحقق أفضل النماذج لدينا درجات بلو من 30.45 ل ill → DE و 38.61 ل De → EN على DataSet IWSLT'14، و 31.26 ل EN → DE و 34.94 ل De → EN على WMT 14 DataSet، مما يتجاوز جميع الأرقام المنشورة.
The success of bidirectional encoders using masked language models, such as BERT, on numerous natural language processing tasks has prompted researchers to attempt to incorporate these pre-trained models into neural machine translation (NMT) systems. However, proposed methods for incorporating pre-trained models are non-trivial and mainly focus on BERT, which lacks a comparison of the impact that other pre-trained models may have on translation performance. In this paper, we demonstrate that simply using the output (contextualized embeddings) of a tailored and suitable bilingual pre-trained language model (dubbed BiBERT) as the input of the NMT encoder achieves state-of-the-art translation performance. Moreover, we also propose a stochastic layer selection approach and a concept of a dual-directional translation model to ensure the sufficient utilization of contextualized embeddings. In the case of without using back translation, our best models achieve BLEU scores of 30.45 for En→De and 38.61 for De→En on the IWSLT'14 dataset, and 31.26 for En→De and 34.94 for De→En on the WMT'14 dataset, which exceeds all published numbers.
References used
https://aclanthology.org/
rate research
Read More
In this work, we analyze the robustness of neural machine translation systems towards grammatical perturbations in the source. In particular, we focus on morphological inflection related perturbations. While this has been recently studied for English
→French (MORPHEUS) (Tan et al., 2020), it is unclear how this extends to Any→English translation systems. We propose MORPHEUS-MULTILINGUAL that utilizes UniMorph dictionaries to identify morphological perturbations to source that adversely affect the translation models. Along with an analysis of state-of-the-art pretrained MT systems, we train and analyze systems for 11 language pairs using the multilingual TED corpus (Qi et al., 2018). We also compare this to actual errors of non-native speakers using Grammatical Error Correction datasets. Finally, we present a qualitative and quantitative analysis of the robustness of Any→English translation systems.
Neural machine translation (NMT) models are data-driven and require large-scale training corpus. In practical applications, NMT models are usually trained on a general domain corpus and then fine-tuned by continuing training on the in-domain corpus.
However, this bears the risk of catastrophic forgetting that the performance on the general domain is decreased drastically. In this work, we propose a new continual learning framework for NMT models. We consider a scenario where the training is comprised of multiple stages and propose a dynamic knowledge distillation technique to alleviate the problem of catastrophic forgetting systematically. We also find that the bias exists in the output linear projection when fine-tuning on the in-domain corpus, and propose a bias-correction module to eliminate the bias. We conduct experiments on three representative settings of NMT application. Experimental results show that the proposed method achieves superior performance compared to baseline models in all settings.
Recent research questions the importance of the dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. In this paper, we push further in this research line and propose a novel substitute
mechanism for self-attention: Recurrent AtteNtion (RAN) . RAN directly learns attention weights without any token-to-token interaction and further improves their capacity by layer-to-layer interaction. Across an extensive set of experiments on 10 machine translation tasks, we find that RAN models are competitive and outperform their Transformer counterpart in certain scenarios, with fewer parameters and inference time. Particularly, when apply RAN to the decoder of Transformer, there brings consistent improvements by about +0.5 BLEU on 6 translation tasks and +1.0 BLEU on Turkish-English translation task. In addition, we conduct extensive analysis on the attention weights of RAN to confirm their reasonableness. Our RAN is a promising alternative to build more effective and efficient NMT models.
Most current neural machine translation models adopt a monotonic decoding order of either left-to-right or right-to-left. In this work, we propose a novel method that breaks up the limitation of these decoding orders, called Smart-Start decoding. Mor
e specifically, our method first predicts a median word. It starts to decode the words on the right side of the median word and then generates words on the left. We evaluate the proposed Smart-Start decoding method on three datasets. Experimental results show that the proposed method can significantly outperform strong baseline models.
We propose a data augmentation method for neural machine translation. It works by interpreting language models and phrasal alignment causally. Specifically, it creates augmented parallel translation corpora by generating (path-specific) counterfactua
l aligned phrases. We generate these by sampling new source phrases from a masked language model, then sampling an aligned counterfactual target phrase by noting that a translation language model can be interpreted as a Gumbel-Max Structural Causal Model (Oberst and Sontag, 2019). Compared to previous work, our method takes both context and alignment into account to maintain the symmetry between source and target sequences. Experiments on IWSLT'15 English → Vietnamese, WMT'17 English → German, WMT'18 English → Turkish, and WMT'19 robust English → French show that the method can improve the performance of translation, backtranslation and translation robustness.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
