Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userInducing Language-Agnostic Multilingual Representations
تحفيز التعليمات متعددة اللغات اللغوية
90
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تمثل التمثيلات المتبادلة القدرة على جعل تقنيات NLP المتاحة للغالبية العظمى من اللغات في العالم. ومع ذلك، فإنهم يتطلبون حاليا كوربورا محدبة كبيرة أو الوصول إلى لغات مماثلة من الناحية النموذجية. في هذا العمل، نتعلم هذه العقبات من خلال إزالة إشارات هوية اللغة من المدينات متعددة اللغات. ندرس ثلاث نهج لذلك: (1) إعادة محاذاة مساحات ناقلات اللغات المستهدفة (كل ذلك) إلى لغة مصدر محوري؛ (2) إزالة الوسائل والفروقة الخاصة باللغة، والتي تفرج بشكل أفضل من التمييز بين المدينين كمنتج ثانوي؛ (3) زيادة تشابه الإدخال عبر اللغات عن طريق إزالة الانقباضات المورفولوجية وإعادة ترتيب الجملة. نقيم تقييم XNLI وتقييم MT بدون مرجع عبر 19 لغة متنوعة من الناحية النموذجية. تعرض نتائجنا قيود هذه الأساليب - على عكس تطبيع ناقلات ومحاذاة الفضاء المتجه وتطبيع النصوص لا يحقق مكاسب متسقة عبر اللوافع واللغات. نظرا لآثاره المضافة للنهج، فإن مزيجها يقلل فجوة النقل عبر اللغات بمقدار 8.9 نقطة (M-Bert) و 18.2 نقطة (XLM-R) في المتوسط عبر جميع المهام واللغات.
Cross-lingual representations have the potential to make NLP techniques available to the vast majority of languages in the world. However, they currently require large pretraining corpora or access to typologically similar languages. In this work, we address these obstacles by removing language identity signals from multilingual embeddings. We examine three approaches for this: (i) re-aligning the vector spaces of target languages (all together) to a pivot source language; (ii) removing language-specific means and variances, which yields better discriminativeness of embeddings as a by-product; and (iii) increasing input similarity across languages by removing morphological contractions and sentence reordering. We evaluate on XNLI and reference-free MT evaluation across 19 typologically diverse languages. Our findings expose the limitations of these approaches---unlike vector normalization, vector space re-alignment and text normalization do not achieve consistent gains across encoders and languages. Due to the approaches' additive effects, their combination decreases the cross-lingual transfer gap by 8.9 points (m-BERT) and 18.2 points (XLM-R) on average across all tasks and languages, however.
References used
https://aclanthology.org/
rate research
Read More
We propose a method to distill a language-agnostic meaning embedding from a multilingual sentence encoder. By removing language-specific information from the original embedding, we retrieve an embedding that fully represents the sentence's meaning. T
he proposed method relies only on parallel corpora without any human annotations. Our meaning embedding allows efficient cross-lingual sentence similarity estimation by simple cosine similarity calculation. Experimental results on both quality estimation of machine translation and cross-lingual semantic textual similarity tasks reveal that our method consistently outperforms the strong baselines using the original multilingual embedding. Our method consistently improves the performance of any pre-trained multilingual sentence encoder, even in low-resource language pairs where only tens of thousands of parallel sentence pairs are available.
Pre-trained multilingual language models have become an important building block in multilingual Natural Language Processing. In the present paper, we investigate a range of such models to find out how well they transfer discourse-level knowledge acr
oss languages. This is done with a systematic evaluation on a broader set of discourse-level tasks than has been previously been assembled. We find that the XLM-RoBERTa family of models consistently show the best performance, by simultaneously being good monolingual models and degrading relatively little in a zero-shot setting. Our results also indicate that model distillation may hurt the ability of cross-lingual transfer of sentence representations, while language dissimilarity at most has a modest effect. We hope that our test suite, covering 5 tasks with a total of 22 languages in 10 distinct families, will serve as a useful evaluation platform for multilingual performance at and beyond the sentence level.
Can pre-trained BERT for one language and GPT for another be glued together to translate texts? Self-supervised training using only monolingual data has led to the success of pre-trained (masked) language models in many NLP tasks. However, directly c
onnecting BERT as an encoder and GPT as a decoder can be challenging in machine translation, for GPT-like models lack a cross-attention component that is needed in seq2seq decoders. In this paper, we propose Graformer to graft separately pre-trained (masked) language models for machine translation. With monolingual data for pre-training and parallel data for grafting training, we maximally take advantage of the usage of both types of data. Experiments on 60 directions show that our method achieves average improvements of 5.8 BLEU in x2en and 2.9 BLEU in en2x directions comparing with the multilingual Transformer of the same size.
Abstract Prior studies in multilingual language modeling (e.g., Cotterell et al., 2018; Mielke et al., 2019) disagree on whether or not inflectional morphology makes languages harder to model. We attempt to resolve the disagreement and extend those s
tudies. We compile a larger corpus of 145 Bible translations in 92 languages and a larger number of typological features.1 We fill in missing typological data for several languages and consider corpus-based measures of morphological complexity in addition to expert-produced typological features. We find that several morphological measures are significantly associated with higher surprisal when LSTM models are trained with BPE-segmented data. We also investigate linguistically motivated subword segmentation strategies like Morfessor and Finite-State Transducers (FSTs) and find that these segmentation strategies yield better performance and reduce the impact of a language's morphology on language modeling.
Pretrained multilingual language models have become a common tool in transferring NLP capabilities to low-resource languages, often with adaptations. In this work, we study the performance, extensibility, and interaction of two such adaptations: voca
bulary augmentation and script transliteration. Our evaluations on part-of-speech tagging, universal dependency parsing, and named entity recognition in nine diverse low-resource languages uphold the viability of these approaches while raising new questions around how to optimally adapt multilingual models to low-resource settings.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
