Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userShould we Stop Training More Monolingual Models, and Simply Use Machine Translation Instead?
يجب أن نتوقف عن تدريب المزيد من نماذج أحادية الأونلينغ، وبما هو استخدام الترجمة الآلية بدلا من ذلك؟
141
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
معظم العمل في NLP يجعل الافتراض أنه من المرغوب فيه تطوير حلول باللغة الأم المعنية. وبالتالي هناك اتجاه قوي نحو بناء نماذج لغات أصلية حتى لغات الموارد المنخفضة. تساهم هذه الورقة في هذا التطور، واستكشف فكرة ترجمة البيانات ببساطة إلى اللغة الإنجليزية، مما يتيح استخدام نماذج اللغة الإنجليزية ذات الاحترام، واسعة النطاق. نوضح تجريبيا أن نموذج اللغة الإنجليزية الكبير إلى جانب الترجمة الآلية الحديثة يتفوقون على نماذج لغة أصلية في معظم اللغات الاسكندنافية. الاستثناء لهذه الفنلندية، والتي نفترض أنها بسبب جودة الترجمة الأدنى. تشير نتائجنا إلى أن الترجمة الآلية هي تقنية ناضجة، تثير حجة مضادة خطيرة لتدريب نماذج اللغة الأم لغات الموارد المنخفضة. لذلك تسعى هذه الورقة إلى اتخاذ نقطة استفزازية ولكنها مهمة. نظرا لأن نماذج اللغة الإنجليزية تتحسن بوتيرة غير مسبوقة، والتي تعمل بدورها على تحسين الترجمة الآلية، فهي من نقطة حاملة تجريبية وبيئية أكثر فعالية لترجمة البيانات من لغات الموارد المنخفضة إلى اللغة الإنجليزية، بدلا من بناء نماذج لغة لهذه اللغات.
Most work in NLP makes the assumption that it is desirable to develop solutions in the native language in question. There is consequently a strong trend towards building native language models even for low-resource languages. This paper questions this development, and explores the idea of simply translating the data into English, thereby enabling the use of pretrained, and large-scale, English language models. We demonstrate empirically that a large English language model coupled with modern machine translation outperforms native language models in most Scandinavian languages. The exception to this is Finnish, which we assume is due to inferior translation quality. Our results suggest that machine translation is a mature technology, which raises a serious counter-argument for training native language models for low-resource languages. This paper therefore strives to make a provocative but important point. As English language models are improving at an unprecedented pace, which in turn improves machine translation, it is from an empirical and environmental stand-point more effective to translate data from low-resource languages into English, than to build language models for such languages.
References used
https://aclanthology.org/
rate research
Read More
Neural machine translation (NMT) models are typically trained using a softmax cross-entropy loss where the softmax distribution is compared against the gold labels. In low-resource scenarios and NMT models tend to perform poorly because the model tra
ining quickly converges to a point where the softmax distribution computed using logits approaches the gold label distribution. Although label smoothing is a well-known solution to address this issue and we further propose to divide the logits by a temperature coefficient greater than one and forcing the softmax distribution to be smoother during training. This makes it harder for the model to quickly over-fit. In our experiments on 11 language pairs in the low-resource Asian Language Treebank dataset and we observed significant improvements in translation quality. Our analysis focuses on finding the right balance of label smoothing and softmax tempering which indicates that they are orthogonal methods. Finally and a study of softmax entropies and gradients reveal the impact of our method on the internal behavior of our NMT models.
Most of the recent Natural Language Processing(NLP) studies are based on the Pretrain-Finetuning Approach (PFA), but in small and medium-sized enterprises or companies with insufficient hardware there are many limitations to servicing NLP application
software using such technology due to slow speed and insufficient memory. The latest PFA technologies require large amounts of data, especially for low-resource languages, making them much more difficult to work with. We propose a new tokenization method, ONE-Piece, to address this limitation that combines the morphology-considered subword tokenization method and the vocabulary method used after probing for an existing method that has not been carefully considered before. Our proposed method can also be used without modifying the model structure. We experiment by applying ONE-Piece to Korean, a morphologically-rich and low-resource language. We derive an optimal subword tokenization result for Korean-English machine translation by conducting a case study that combines the subword tokenization method, morphological segmentation, and vocabulary method. Through comparative experiments with all the tokenization methods currently used in NLP research, ONE-Piece achieves performance comparable to the current Korean-English machine translation state-of-the-art model.
The MultiTraiNMT Erasmus+ project aims at developing an open innovative syllabus in neural machine translation (NMT) for language learners and translators as multilingual citizens. Machine translation is seen as a resource that can support citizens i
n their attempt to acquire and develop language skills if they are trained in an informed and critical way. Machine translation could thus help tackle the mismatch between the desired EU aim of having multilingual citizens who speak at least two foreign languages and the current situation in which citizens generally fall far short of this objective. The training materials consists of an open-access coursebook, an open-source NMT web application called MutNMT for training purposes, and corresponding activities.
Latent alignment objectives such as CTC and AXE significantly improve non-autoregressive machine translation models. Can they improve autoregressive models as well? We explore the possibility of training autoregressive machine translation models with
latent alignment objectives, and observe that, in practice, this approach results in degenerate models. We provide a theoretical explanation for these empirical results, and prove that latent alignment objectives are incompatible with teacher forcing.
Identification of lexical borrowings, transfer of words between languages, is an essential practice of historical linguistics and a vital tool in analysis of language contact and cultural events in general. We seek to improve tools for automatic dete
ction of lexical borrowings, focusing here on detecting borrowed words from monolingual wordlists. Starting with a recurrent neural lexical language model and competing entropies approach, we incorporate a more current Transformer based lexical model. From there we experiment with several different models and approaches including a lexical donor model with augmented wordlist. The Transformer model reduces execution time and minimally improves borrowing detection. The augmented donor model shows some promise. A substantive change in approach or model is needed to make significant gains in identification of lexical borrowings.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
