Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userPruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation
نموذج تشذيب ثم توسيع نموذج لتكييف المجال من الترجمة الآلية العصبية
231
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
يستخدم تكيف المجال على نطاق واسع في التطبيقات العملية للترجمة الآلية العصبية، والتي تهدف إلى تحقيق أداء جيد على كل من المجال العام والبيانات داخل المجال. ومع ذلك، فإن الأساليب الحالية لتكييف المجال عادة ما تعاني من النسيان الكارثي، والاختلاف المجال الكبير، والانفجار النموذجي. لمعالجة هذه المشكلات الثلاثة، نقترح طريقة للتقسيم والتغلب عليها "والتي تعتمد على أهمية الخلايا العصبية أو المعلمات لنموذج الترجمة. في هذه الطريقة، نقوم أولا بإزالة النموذج ويحافظ على الخلايا العصبية أو المعلمات المهمة فقط، مما يجعلها مسؤولة عن كل من المجال العام والترجمة داخل المجال. ثم علينا مزيد من تدريب النموذج المعاني الذي يشرف عليه النموذج الكامل الأصلي مع تقطير المعرفة. أخيرا، نوسع النموذج إلى الحجم الأصلي وضبط المعلمات المضافة للترجمة داخل المجال. أجرينا تجارب على أزواج ومجالات مختلفة للغة والنتائج تظهر أن طريقتنا يمكن أن تحقق تحسينات كبيرة مقارنة بالعديد من خطوط الأساس القوية.
Domain Adaptation is widely used in practical applications of neural machine translation, which aims to achieve good performance on both general domain and in-domain data. However, the existing methods for domain adaptation usually suffer from catastrophic forgetting, large domain divergence, and model explosion. To address these three problems, we propose a method of divide and conquer'' which is based on the importance of neurons or parameters for the translation model. In this method, we first prune the model and only keep the important neurons or parameters, making them responsible for both general-domain and in-domain translation. Then we further train the pruned model supervised by the original whole model with knowledge distillation. Last we expand the model to the original size and fine-tune the added parameters for the in-domain translation. We conducted experiments on different language pairs and domains and the results show that our method can achieve significant improvements compared with several strong baselines.
References used
https://aclanthology.org/
rate research
Read More
We study the problem of domain adaptation in Neural Machine Translation (NMT) when domain-specific data cannot be shared due to confidentiality or copyright issues. As a first step, we propose to fragment data into phrase pairs and use a random sampl
e to fine-tune a generic NMT model instead of the full sentences. Despite the loss of long segments for the sake of confidentiality protection, we find that NMT quality can considerably benefit from this adaptation, and that further gains can be obtained with a simple tagging technique.
Translation quality can be improved by global information from the required target sentence because the decoder can understand both past and future information. However, the model needs additional cost to produce and consider such global information.
In this work, to inject global information but also save cost, we present an efficient method to sample and consider a semantic draft as global information from semantic space for decoding with almost free of cost. Unlike other successful adaptations, we do not have to perform an EM-like process that repeatedly samples a possible semantic from the semantic space. Empirical experiments show that the presented method can achieve competitive performance in common language pairs with a clear advantage in inference efficiency. We will open all our source code on GitHub.
Abstract Recent improvements in the predictive quality of natural language processing systems are often dependent on a substantial increase in the number of model parameters. This has led to various attempts of compressing such models, but existing m
ethods have not considered the differences in the predictive power of various model components or in the generalizability of the compressed models. To understand the connection between model compression and out-of-distribution generalization, we define the task of compressing language representation models such that they perform best in a domain adaptation setting. We choose to address this problem from a causal perspective, attempting to estimate the average treatment effect (ATE) of a model component, such as a single layer, on the model's predictions. Our proposed ATE-guided Model Compression scheme (AMoC), generates many model candidates, differing by the model components that were removed. Then, we select the best candidate through a stepwise regression model that utilizes the ATE to predict the expected performance on the target domain. AMoC outperforms strong baselines on dozens of domain pairs across three text classification and sequence tagging tasks.1
This paper considers the unsupervised domain adaptation problem for neural machine translation (NMT), where we assume the access to only monolingual text in either the source or target language in the new domain. We propose a cross-lingual data selec
tion method to extract in-domain sentences in the missing language side from a large generic monolingual corpus. Our proposed method trains an adaptive layer on top of multilingual BERT by contrastive learning to align the representation between the source and target language. This then enables the transferability of the domain classifier between the languages in a zero-shot manner. Once the in-domain data is detected by the classifier, the NMT model is then adapted to the new domain by jointly learning translation and domain discrimination tasks. We evaluate our cross-lingual data selection method on NMT across five diverse domains in three language pairs, as well as a real-world scenario of translation for COVID-19. The results show that our proposed method outperforms other selection baselines up to +1.5 BLEU score.
Production NMT systems typically need to serve niche domains that are not covered by adequately large and readily available parallel corpora. As a result, practitioners often fine-tune general purpose models to each of the domains their organisation
caters to. The number of domains however can often become large, which in combination with the number of languages that need serving can lead to an unscalable fleet of models to be developed and maintained. We propose Multi Dimensional Tagging, a method for fine-tuning a single NMT model on several domains simultaneously, thus drastically reducing development and maintenance costs. We run experiments where a single MDT model compares favourably to a set of SOTA specialist models, even when evaluated on the domain those baselines have been fine-tuned on. Besides BLEU, we report human evaluation results. MDT models are now live at Booking.com, powering an MT engine that serves millions of translations a day in over 40 different languages.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
