Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userWhen Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models
عندما تكون غير مرئية من MBERT هي مجرد بداية: معالجة لغات جديدة مع نماذج لغة متعددة اللغات
343
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
أصبح التحويل التعلم بناء على نماذج لغة المحترفين على كمية كبيرة من البيانات الخام نموذجا جديدا للوصول إلى الأداء الحديث في NLP. ومع ذلك، لا يزال من غير الواضح كيف ينبغي تطبيق هذا النهج لغات غير مرئية غير مشمولة بأي نموذج لغوي متعدد اللغات واسعة ناتجا، والذي يتم توفير كمية صغيرة فقط من البيانات الخام فقط. في هذا العمل، من خلال مقارنة النماذج متعددة اللغات وأنتغوية، نوضح أن هذه النماذج تتصرف بطرق متعددة على اللغات غير المرئية. تستفيد بعض اللغات بشكل كبير من تعلم التعلم والتصرف بالمثل إلى لغات موارد عالية مرتبطة ارتباطا وثيقا في حين أن الآخرين على ما يبدو لا. التركيز على الأخير، نظرا لأن هذا الفشل في النقل يرتبط إلى حد كبير بتأثير البرنامج النصي المستخدم لكتابة هذه اللغات. نظهر أن ترجمة هذه اللغات تعمل بشكل كبير على تحسين إمكانات نماذج اللغة متعددة اللغات على نطاق واسع في مهام المصب. توفر هذه النتيجة اتجاها واعدا نحو جعل هذه النماذج متعددة اللغات بشكل كبير مفيدة لمجموعة جديدة من اللغات غير المرئية.
Transfer learning based on pretraining language models on a large amount of raw data has become a new norm to reach state-of-the-art performance in NLP. Still, it remains unclear how this approach should be applied for unseen languages that are not covered by any available large-scale multilingual language model and for which only a small amount of raw data is generally available. In this work, by comparing multilingual and monolingual models, we show that such models behave in multiple ways on unseen languages. Some languages greatly benefit from transfer learning and behave similarly to closely related high resource languages whereas others apparently do not. Focusing on the latter, we show that this failure to transfer is largely related to the impact of the script used to write such languages. We show that transliterating those languages significantly improves the potential of large-scale multilingual language models on downstream tasks. This result provides a promising direction towards making these massively multilingual models useful for a new set of unseen languages.
References used
https://aclanthology.org/
rate research
Read More
Pre-trained multilingual language models have become an important building block in multilingual Natural Language Processing. In the present paper, we investigate a range of such models to find out how well they transfer discourse-level knowledge acr
oss languages. This is done with a systematic evaluation on a broader set of discourse-level tasks than has been previously been assembled. We find that the XLM-RoBERTa family of models consistently show the best performance, by simultaneously being good monolingual models and degrading relatively little in a zero-shot setting. Our results also indicate that model distillation may hurt the ability of cross-lingual transfer of sentence representations, while language dissimilarity at most has a modest effect. We hope that our test suite, covering 5 tasks with a total of 22 languages in 10 distinct families, will serve as a useful evaluation platform for multilingual performance at and beyond the sentence level.
Masked language models have quickly become the de facto standard when processing text. Recently, several approaches have been proposed to further enrich word representations with external knowledge sources such as knowledge graphs. However, these mod
els are devised and evaluated in a monolingual setting only. In this work, we propose a language-independent entity prediction task as an intermediate training procedure to ground word representations on entity semantics and bridge the gap across different languages by means of a shared vocabulary of entities. We show that our approach effectively injects new lexical-semantic knowledge into neural models, improving their performance on different semantic tasks in the zero-shot crosslingual setting. As an additional advantage, our intermediate training does not require any supplementary input, allowing our models to be applied to new datasets right away. In our experiments, we use Wikipedia articles in up to 100 languages and already observe consistent gains compared to strong baselines when predicting entities using only the English Wikipedia. Further adding extra languages lead to improvements in most tasks up to a certain point, but overall we found it non-trivial to scale improvements in model transferability by training on ever increasing amounts of Wikipedia languages.
Pretrained multilingual language models have become a common tool in transferring NLP capabilities to low-resource languages, often with adaptations. In this work, we study the performance, extensibility, and interaction of two such adaptations: voca
bulary augmentation and script transliteration. Our evaluations on part-of-speech tagging, universal dependency parsing, and named entity recognition in nine diverse low-resource languages uphold the viability of these approaches while raising new questions around how to optimally adapt multilingual models to low-resource settings.
We analyze if large language models are able to predict patterns of human reading behavior. We compare the performance of language-specific and multilingual pretrained transformer models to predict reading time measures reflecting natural human sente
nce processing on Dutch, English, German, and Russian texts. This results in accurate models of human reading behavior, which indicates that transformer models implicitly encode relative importance in language in a way that is comparable to human processing mechanisms. We find that BERT and XLM models successfully predict a range of eye tracking features. In a series of experiments, we analyze the cross-domain and cross-language abilities of these models and show how they reflect human sentence processing.
The most successful approach to Neural Machine Translation (NMT) when only monolingual training data is available, called unsupervised machine translation, is based on back-translation where noisy translations are generated to turn the task into a su
pervised one. However, back-translation is computationally very expensive and inefficient. This work explores a novel, efficient approach to unsupervised NMT. A transformer, initialized with cross-lingual language model weights, is fine-tuned exclusively on monolingual data of the target language by jointly learning on a paraphrasing and denoising autoencoder objective. Experiments are conducted on WMT datasets for German-English, French-English, and Romanian-English. Results are competitive to strong baseline unsupervised NMT models, especially for closely related source languages (German) compared to more distant ones (Romanian, French), while requiring about a magnitude less training time.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
