Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userThe Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation
BIND مزدوج الموارد المنخفضة: دراسة تجريبية للتذكيرية لترجمة آلات الموارد المنخفضة
286
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
أصبح أكبر انفجار أفضل في عدد المعلمات في الشبكات العصبية العميقة جعلته صعبة بشكل متزايد لجعل الشبكات الحديثة التي يمكن الوصول إليها في البيئات المقيدة لحسابها. أخذت تقنيات ضغط الأهمية المتجددة كوسيلة لسد الفجوة. ومع ذلك، فإن تقييم المفاضلات المتكبدة من خلال تقنيات الضغط الشعبية قد تركزت على مجموعات بيانات عالية الموارد. في هذا العمل، نعتبر بدلا من ذلك تأثير الضغط في نظام محدود من البيانات. نقدم مصطلح رابط مزدوج الموارد المنخفضة للإشارة إلى حدوث قيود البيانات ويحسب قيود الموارد. هذا هو إعداد شائع لبرنامج NLP لغات الموارد المنخفضة، ومع ذلك، تتم دراسة المفاضلات في الأداء بشكل سيء. يقدم عملنا رؤى مفاجئة في العلاقة بين القدرات والتعميم في الأنظمة المحدودة البيانات لمهمة الترجمة الآلية. تجاربنا على حجم تشذيب الترجمات من الإنجليزية إلى يوروبا، هاوسا، إغيبو وألماني تظهر أنه في أنظمة الموارد المنخفضة، تحافظ Sparsity على أداء على جمل متكررة ولكن لها تأثير متباين على النادر. ومع ذلك، فإنه يعمل على تحسين نوبات التوزيع، وخاصة بالنسبة لمجموعات البيانات المميزة للغاية عن توزيع التدريب. تشير نتائجنا إلى أن Sparsity يمكن أن تلعب دورا مفيدا في الحد من حفظ سمات التردد المنخفضة، وبالتالي يقدم حلا واعدا للربط المزدوج الموارد المنخفضة.
A bigger is better'' explosion in the number of parameters in deep neural networks has made it increasingly challenging to make state-of-the-art networks accessible in compute-restricted environments. Compression techniques have taken on renewed importance as a way to bridge the gap. However, evaluation of the trade-offs incurred by popular compression techniques has been centered on high-resource datasets. In this work, we instead consider the impact of compression in a data-limited regime. We introduce the term low-resource double bind to refer to the co-occurrence of data limitations and compute resource constraints. This is a common setting for NLP for low-resource languages, yet the trade-offs in performance are poorly studied. Our work offers surprising insights into the relationship between capacity and generalization in data-limited regimes for the task of machine translation. Our experiments on magnitude pruning for translations from English into Yoruba, Hausa, Igbo and German show that in low-resource regimes, sparsity preserves performance on frequent sentences but has a disparate impact on infrequent ones. However, it improves robustness to out-of-distribution shifts, especially for datasets that are very distinct from the training distribution. Our findings suggest that sparsity can play a beneficial role at curbing memorization of low frequency attributes, and therefore offers a promising solution to the low-resource double bind.
References used
https://aclanthology.org/
rate research
Read More
In this work, we investigate methods for the challenging task of translating between low- resource language pairs that exhibit some level of similarity. In particular, we consider the utility of transfer learning for translating between several Indo-
European low-resource languages from the Germanic and Romance language families. In particular, we build two main classes of transfer-based systems to study how relatedness can benefit the translation performance. The primary system fine-tunes a model pre-trained on a related language pair and the contrastive system fine-tunes one pre-trained on an unrelated language pair. Our experiments show that although relatedness is not necessary for transfer learning to work, it does benefit model performance.
In this paper and we explore different techniques of overcoming the challenges of low-resource in Neural Machine Translation (NMT) and specifically focusing on the case of English-Marathi NMT. NMT systems require a large amount of parallel corpora to
obtain good quality translations. We try to mitigate the low-resource problem by augmenting parallel corpora or by using transfer learning. Techniques such as Phrase Table Injection (PTI) and back-translation and mixing of language corpora are used for enhancing the parallel data; whereas pivoting and multilingual embeddings are used to leverage transfer learning. For pivoting and Hindi comes in as assisting language for English-Marathi translation. Compared to baseline transformer model and a significant improvement trend in BLEU score is observed across various techniques. We have done extensive manual and automatic and qualitative evaluation of our systems. Since the trend in Machine Translation (MT) today is post-editing and measuring of Human Effort Reduction (HER) and we have given our preliminary observations on Translation Edit Rate (TER) vs. BLEU score study and where TER is regarded as a measure of HER.
Low-resource languages can be understood as languages that are more scarce, less studied, less privileged, less commonly taught and for which there are less resources available (Singh, 2008; Cieri et al., 2016; Magueresse et al., 2020). Natural Langu
age Processing (NLP) research and technology mainly focuses on those languages for which there are large data sets available. To illustrate differences in data availability: there are 6 million Wikipedia articles available for English, 2 million for Dutch, and merely 82 thousand for Albanian. The scarce data issue becomes increasingly apparent when large parallel data sets are required for applications such as Neural Machine Translation (NMT). In this work, we investigate to what extent translation between Albanian (SQ) and Dutch (NL) is possible comparing a one-to-one (SQ↔AL) model, a low-resource pivot-based approach (English (EN) as pivot) and a zero-shot translation (ZST) (Johnson et al., 2016; Mattoni et al., 2017) system. From our experiments, it results that the EN-pivot-model outperforms both the direct one-to-one and the ZST model. Since often, small amounts of parallel data are available for low-resource languages or settings, experiments were conducted using small sets of parallel NL↔SQ data. The ZST appeared to be the worst performing models. Even when the available parallel data (NL↔SQ) was added, i.e. in a few-shot setting (FST), it remained the worst performing system according to the automatic (BLEU and TER) and human evaluation.
Neural relation extraction models have shown promising results in recent years; however, the model performance drops dramatically given only a few training samples. Recent works try leveraging the advance in few-shot learning to solve the low resourc
e problem, where they train label-agnostic models to directly compare the semantic similarities among context sentences in the embedding space. However, the label-aware information, i.e., the relation label that contains the semantic knowledge of the relation itself, is often neglected for prediction. In this work, we propose a framework considering both label-agnostic and label-aware semantic mapping information for low resource relation extraction. We show that incorporating the above two types of mapping information in both pretraining and fine-tuning can significantly improve the model performance on low-resource relation extraction tasks.
State-of-the-art abstractive summarization models generally rely on extensive labeled data, which lowers their generalization ability on domains where such data are not available. In this paper, we present a study of domain adaptation for the abstrac
tive summarization task across six diverse target domains in a low-resource setting. Specifically, we investigate the second phase of pre-training on large-scale generative models under three different settings: 1) source domain pre-training; 2) domain-adaptive pre-training; and 3) task-adaptive pre-training. Experiments show that the effectiveness of pre-training is correlated with the similarity between the pre-training data and the target domain task. Moreover, we find that continuing pre-training could lead to the pre-trained model's catastrophic forgetting, and a learning method with less forgetting can alleviate this issue. Furthermore, results illustrate that a huge gap still exists between the low-resource and high-resource settings, which highlights the need for more advanced domain adaptation methods for the abstractive summarization task.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
