Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userStructural Biases for Improving Transformers on Translation into Morphologically Rich Languages
التحيزات الهيكلية لتحسين المحولات على الترجمة إلى لغات غنية مورفولوجية
135
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
شهدت الترجمة الآلية التقدم السريع مع ظهور النماذج القائمة على المحولات. لا تحتوي هذه النماذج على هيكل لغوي صريح مبني عليهم، ومع ذلك فقد لا يزالون لا يزالون يتعلمون ضمنيا علاقات منظمة من خلال حضور الرموز ذات الصلة. نحن نفترض أن هذا التعلم الهيكلي أكثر قوة من خلال مرفق المحولات بشكل صريح مع التحيز الهيكلية، ونحن نحقق في طريقتين للبناء في مثل هذا التحيز. طريقة واحدة، TP-Transformer، تزيد من بنية المحولات التقليدية لتضمين مكون إضافي لتمثيل الهيكل. الطريقة الثانية تغلق الهيكل على مستوى البيانات عن طريق تجزئة البيانات مع التوت التمييز المورفولوجي. نختبر هذه الأساليب على الترجمة من الإنجليزية إلى لغات غنية مورفولوجية، التركية والأجنبية، والنظر في المقاييس التلقائية والتقييمات البشرية. نجد أن كل هذين من النهجين يسمح للشبكة لتحقيق أداء أفضل، ولكن هذا التحسين يعتمد على حجم مجموعة البيانات. باختصار طرق الترميز الهيكلية تجعل المحولات أكثر كفاءة عينة، مما يتيح لهم الأداء بشكل أفضل من كميات أصغر من البيانات.
Machine translation has seen rapid progress with the advent of Transformer-based models. These models have no explicit linguistic structure built into them, yet they may still implicitly learn structured relationships by attending to relevant tokens. We hypothesize that this structural learning could be made more robust by explicitly endowing Transformers with a structural bias, and we investigate two methods for building in such a bias. One method, the TP-Transformer, augments the traditional Transformer architecture to include an additional component to represent structure. The second method imbues structure at the data level by segmenting the data with morphological tokenization. We test these methods on translating from English into morphologically rich languages, Turkish and Inuktitut, and consider both automatic metrics and human evaluations. We find that each of these two approaches allows the network to achieve better performance, but this improvement is dependent on the size of the dataset. In sum, structural encoding methods make Transformers more sample-efficient, enabling them to perform better from smaller amounts of data.
References used
https://aclanthology.org/
rate research
Read More
Dravidian languages, such as Kannada and Tamil, are notoriously difficult to translate by state-of-the-art neural models. This stems from the fact that these languages are morphologically very rich as well as being low-resourced. In this paper, we fo
cus on subword segmentation and evaluate Linguistically Motivated Vocabulary Reduction (LMVR) against the more commonly used SentencePiece (SP) for the task of translating from English into four different Dravidian languages. Additionally we investigate the optimal subword vocabulary size for each language. We find that SP is the overall best choice for segmentation, and that larger dictionary sizes lead to higher translation quality.
Neural Machine Translation (NMT) for Low Resource Languages (LRL) is often limited by the lack of available training data, making it necessary to explore additional techniques to improve translation quality. We propose the use of the Prefix-Root-Post
fix-Encoding (PRPE) subword segmentation algorithm to improve translation quality for LRLs, using two agglutinative languages as case studies: Quechua and Indonesian. During the course of our experiments, we reintroduce a parallel corpus for Quechua-Spanish translation that was previously unavailable for NMT. Our experiments show the importance of appropriate subword segmentation, which can go as far as improving translation quality over systems trained on much larger quantities of data. We show this by achieving state-of-the-art results for both languages, obtaining higher BLEU scores than large pre-trained models with much smaller amounts of data.
This paper reports the Machine Translation (MT) systems submitted by the IIITT team for the English→Marathi and English⇔Irish language pairs LoResMT 2021 shared task. The task focuses on getting exceptional translations for rather low-resourced langu
ages like Irish and Marathi. We fine-tune IndicTrans, a pretrained multilingual NMT model for English→Marathi, using external parallel corpus as input for additional training. We have used a pretrained Helsinki-NLP Opus MT English⇔Irish model for the latter language pair. Our approaches yield relatively promising results on the BLEU metrics. Under the team name IIITT, our systems ranked 1, 1, and 2 in English→Marathi, Irish→English, and English→Irish respectively. The codes for our systems are published1 .
For most language combinations and parallel data is either scarce or simply unavailable. To address this and unsupervised machine translation (UMT) exploits large amounts of monolingual data by using synthetic data generation techniques such as back-
translation and noising and while self-supervised NMT (SSNMT) identifies parallel sentences in smaller comparable data and trains on them. To this date and the inclusion of UMT data generation techniques in SSNMT has not been investigated. We show that including UMT techniques into SSNMT significantly outperforms SSNMT (up to +4.3 BLEU and af2en) as well as statistical (+50.8 BLEU) and hybrid UMT (+51.5 BLEU) baselines on related and distantly-related and unrelated language pairs.
Text variational autoencoders (VAEs) are notorious for posterior collapse, a phenomenon where the model's decoder learns to ignore signals from the encoder. Because posterior collapse is known to be exacerbated by expressive decoders, Transformers ha
ve seen limited adoption as components of text VAEs. Existing studies that incorporate Transformers into text VAEs (Li et al., 2020; Fang et al., 2021) mitigate posterior collapse using massive pretraining, a technique unavailable to most of the research community without extensive computing resources. We present a simple two-phase training scheme to convert a sequence-to-sequence Transformer into a VAE with just finetuning. The resulting language model is competitive with massively pretrained Transformer-based VAEs in some internal metrics while falling short on others. To facilitate training we comprehensively explore the impact of common posterior collapse alleviation techniques in the literature. We release our code for reproducability.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
