Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userGeneralised Unsupervised Domain Adaptation of Neural Machine Translation with Cross-Lingual Data Selection
تكيف المجال غير المقترح للتعميم من الترجمة الآلية العصبية مع اختيار البيانات عبر اللغات
604
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تعتبر هذه الورقة مشكلة تكيف المجال غير المدعومة من أجل الترجمة الآلية العصبية (NMT)، حيث نفترض الوصول إلى نص أحادي فقط إما في المصدر أو اللغة المستهدفة في المجال الجديد. نقترح طريقة اختيار البيانات عبر اللغات لاستخراج الجمل داخل المجال في جانب اللغة المفقودة من كوربوس أحادية الأجل عام كبيرة. تقوم طريقةنا المقترحة بتدريب طبقة تكيفية على رأس بيرتف متعدد اللغات من خلال التعلم المتعرج عن تعايز التمثيل بين المصدر واللغة المستهدفة. ثم يتيح ذلك تحويل قابلية تحويل المجال بين اللغات بطريقة طلقة صفرية. بمجرد اكتشاف البيانات داخل المجال من قبل المصنف، يتم بعد ذلك تكييف نموذج NMT بالمجال الجديد من خلال مهام الترجمة التعلم المشتركة ومهام التمييز بين المجال. نقيم طريقة اختيار بياناتنا عبر اللغات لدينا على NMT عبر خمسة مجالات متنوعة في ثلاث أزواج لغوية، وكذلك سيناريو في العالم الحقيقي للترجمة Covid-19. تظهر النتائج أن أسلوبنا المقترح تتفوق على خطوط خطوط خطوط اختيار الاختيار الأخرى تصل إلى +1.5 درجة بلو.
This paper considers the unsupervised domain adaptation problem for neural machine translation (NMT), where we assume the access to only monolingual text in either the source or target language in the new domain. We propose a cross-lingual data selection method to extract in-domain sentences in the missing language side from a large generic monolingual corpus. Our proposed method trains an adaptive layer on top of multilingual BERT by contrastive learning to align the representation between the source and target language. This then enables the transferability of the domain classifier between the languages in a zero-shot manner. Once the in-domain data is detected by the classifier, the NMT model is then adapted to the new domain by jointly learning translation and domain discrimination tasks. We evaluate our cross-lingual data selection method on NMT across five diverse domains in three language pairs, as well as a real-world scenario of translation for COVID-19. The results show that our proposed method outperforms other selection baselines up to +1.5 BLEU score.
References used
https://aclanthology.org/
rate research
Read More
We study the problem of domain adaptation in Neural Machine Translation (NMT) when domain-specific data cannot be shared due to confidentiality or copyright issues. As a first step, we propose to fragment data into phrase pairs and use a random sampl
e to fine-tune a generic NMT model instead of the full sentences. Despite the loss of long segments for the sake of confidentiality protection, we find that NMT quality can considerably benefit from this adaptation, and that further gains can be obtained with a simple tagging technique.
Previous work mainly focuses on improving cross-lingual transfer for NLU tasks with a multilingual pretrained encoder (MPE), or improving the performance on supervised machine translation with BERT. However, it is under-explored that whether the MPE
can help to facilitate the cross-lingual transferability of NMT model. In this paper, we focus on a zero-shot cross-lingual transfer task in NMT. In this task, the NMT model is trained with parallel dataset of only one language pair and an off-the-shelf MPE, then it is directly tested on zero-shot language pairs. We propose SixT, a simple yet effective model for this task. SixT leverages the MPE with a two-stage training schedule and gets further improvement with a position disentangled encoder and a capacity-enhanced decoder. Using this method, SixT significantly outperforms mBART, a pretrained multilingual encoder-decoder model explicitly designed for NMT, with an average improvement of 7.1 BLEU on zero-shot any-to-English test sets across 14 source languages. Furthermore, with much less training computation cost and training data, our model achieves better performance on 15 any-to-English test sets than CRISS and m2m-100, two strong multilingual NMT baselines.
Machine translation usually relies on parallel corpora to provide parallel signals for training. The advent of unsupervised machine translation has brought machine translation away from this reliance, though performance still lags behind traditional
supervised machine translation. In unsupervised machine translation, the model seeks symmetric language similarities as a source of weak parallel signal to achieve translation. Chomsky's Universal Grammar theory postulates that grammar is an innate form of knowledge to humans and is governed by universal principles and constraints. Therefore, in this paper, we seek to leverage such shared grammar clues to provide more explicit language parallel signals to enhance the training of unsupervised machine translation models. Through experiments on multiple typical language pairs, we demonstrate the effectiveness of our proposed approaches.
Domain Adaptation is widely used in practical applications of neural machine translation, which aims to achieve good performance on both general domain and in-domain data. However, the existing methods for domain adaptation usually suffer from catast
rophic forgetting, large domain divergence, and model explosion. To address these three problems, we propose a method of divide and conquer'' which is based on the importance of neurons or parameters for the translation model. In this method, we first prune the model and only keep the important neurons or parameters, making them responsible for both general-domain and in-domain translation. Then we further train the pruned model supervised by the original whole model with knowledge distillation. Last we expand the model to the original size and fine-tune the added parameters for the in-domain translation. We conducted experiments on different language pairs and domains and the results show that our method can achieve significant improvements compared with several strong baselines.
In this paper, we describe our MiSS system that participated in the WMT21 news translation task. We mainly participated in the evaluation of the three translation directions of English-Chinese and Japanese-English translation tasks. In the systems su
bmitted, we primarily considered wider networks, deeper networks, relative positional encoding, and dynamic convolutional networks in terms of model structure, while in terms of training, we investigated contrastive learning-reinforced domain adaptation, self-supervised training, and optimization objective switching training methods. According to the final evaluation results, a deeper, wider, and stronger network can improve translation performance in general, yet our data domain adaption method can improve performance even more. In addition, we found that switching to the use of our proposed objective during the finetune phase using relatively small domain-related data can effectively improve the stability of the model's convergence and achieve better optimal performance.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
