Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userRecurrent Attention for Neural Machine Translation
الاهتمام المتكرر للترجمة الآلية العصبية
348
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
أسئلة البحث الحديثة أهمية الاهتمام الذاتي لمنتج المنتج في نماذج المحولات ويظهر أن معظم رؤساء الاهتمام تعلم أنماطا موضعية بسيطة. في هذه الورقة، ندفع أبعد من ذلك في خط البحث هذا واقتراح آلية بديلة جديدة عن النفس: الاهتمام المتكرر (ران). تتعلم RAN بشكل مباشر أوزان الاهتمام دون أي تفاعل رمزي إلى رمز ويحسن قدرتها على تفاعل الطبقة إلى الطبقة. عبر مجموعة واسعة من التجارب في 10 مهام ترجمة آلية، نجد أن نماذج RAN تنافسية وتفوق نظيرها المحول في بعض السيناريوهات، مع عدد أقل من المعلمات ووقت الاستدلال. خاصة، عند تطبيق ركض إلى فك ترميز المحولات، يجلب التحسينات المتسقة عن طريق حوالي +0.5 بلو في 6 مهام الترجمة و +1.0 Bleu على مهمة الترجمة التركية الإنجليزية. بالإضافة إلى ذلك، نجرينا تحليلا مكثفا بشأن أوزان الاهتمام في ركض لتأكيد المعقولية. ران لدينا هو بديل واعد لبناء نماذج NMT أكثر فعالية وكفاءة.
Recent research questions the importance of the dot-product self-attention in Transformer models and shows that most attention heads learn simple positional patterns. In this paper, we push further in this research line and propose a novel substitute mechanism for self-attention: Recurrent AtteNtion (RAN) . RAN directly learns attention weights without any token-to-token interaction and further improves their capacity by layer-to-layer interaction. Across an extensive set of experiments on 10 machine translation tasks, we find that RAN models are competitive and outperform their Transformer counterpart in certain scenarios, with fewer parameters and inference time. Particularly, when apply RAN to the decoder of Transformer, there brings consistent improvements by about +0.5 BLEU on 6 translation tasks and +1.0 BLEU on Turkish-English translation task. In addition, we conduct extensive analysis on the attention weights of RAN to confirm their reasonableness. Our RAN is a promising alternative to build more effective and efficient NMT models.
References used
https://aclanthology.org/
rate research
Read More
We present a simple method for extending transformers to source-side trees. We define a number of masks that limit self-attention based on relationships among tree nodes, and we allow each attention head to learn which mask or masks to use. On transl
ation from English to various low-resource languages, and translation in both directions between English and German, our method always improves over simple linearization of the source-side parse tree and almost always improves over a sequence-to-sequence baseline, by up to +2.1 BLEU.
Cross-attention is an important component of neural machine translation (NMT), which is always realized by dot-product attention in previous methods. However, dot-product attention only considers the pair-wise correlation between words, resulting in
dispersion when dealing with long sentences and neglect of source neighboring relationships. Inspired by linguistics, the above issues are caused by ignoring a type of cross-attention, called concentrated attention, which focuses on several central words and then spreads around them. In this work, we apply Gaussian Mixture Model (GMM) to model the concentrated attention in cross-attention. Experiments and analyses we conducted on three datasets show that the proposed method outperforms the baseline and has significant improvement on alignment quality, N-gram accuracy, and long sentence translation.
Neural machine translation (NMT) models are data-driven and require large-scale training corpus. In practical applications, NMT models are usually trained on a general domain corpus and then fine-tuned by continuing training on the in-domain corpus.
However, this bears the risk of catastrophic forgetting that the performance on the general domain is decreased drastically. In this work, we propose a new continual learning framework for NMT models. We consider a scenario where the training is comprised of multiple stages and propose a dynamic knowledge distillation technique to alleviate the problem of catastrophic forgetting systematically. We also find that the bias exists in the output linear projection when fine-tuning on the in-domain corpus, and propose a bias-correction module to eliminate the bias. We conduct experiments on three representative settings of NMT application. Experimental results show that the proposed method achieves superior performance compared to baseline models in all settings.
Most current neural machine translation models adopt a monotonic decoding order of either left-to-right or right-to-left. In this work, we propose a novel method that breaks up the limitation of these decoding orders, called Smart-Start decoding. Mor
e specifically, our method first predicts a median word. It starts to decode the words on the right side of the median word and then generates words on the left. We evaluate the proposed Smart-Start decoding method on three datasets. Experimental results show that the proposed method can significantly outperform strong baseline models.
We propose a data augmentation method for neural machine translation. It works by interpreting language models and phrasal alignment causally. Specifically, it creates augmented parallel translation corpora by generating (path-specific) counterfactua
l aligned phrases. We generate these by sampling new source phrases from a masked language model, then sampling an aligned counterfactual target phrase by noting that a translation language model can be interpreted as a Gumbel-Max Structural Causal Model (Oberst and Sontag, 2019). Compared to previous work, our method takes both context and alignment into account to maintain the symmetry between source and target sequences. Experiments on IWSLT'15 English → Vietnamese, WMT'17 English → German, WMT'18 English → Turkish, and WMT'19 robust English → French show that the method can improve the performance of translation, backtranslation and translation robustness.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
