Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userLearning Hard Retrieval Decoder Attention for Transformers
تعلم رموز فك التراجع الصعب الانتباه للمحولات
227
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
يعتمد نموذج الترجمة المحول على آلية الاهتمام المتعدد الرأس، والتي يمكن توازتها بسهولة.تقوم شبكة الاهتمام المتعددة بالاهتمام بأداء وظيفة اهتمام المنتج DOT-Product المعزز بالتوازي، مما تمكن من تمكين النموذج من خلال حضور المعلومات المشتركة إلى معلومات من مختلف الفئات الفرعية التمثيلية في مواقف مختلفة.في هذه الورقة، نقدم نهجا لتعلم اهتمام استرجاع صعب حيث يحضر رأس الاهتمام فقط إلى رمز واحد في الجملة بدلا من جميع الرموز.وبالتالي، يمكن استبدال مضاعفة المصفوفة بين احتمالات الاهتمام وتسلسل القيمة في إيلاء اهتمام منتجات DOT-Product القياسية القياسية بعملية استرجاع بسيطة وفعالة.نظظ أن آلية اهتمام استرجاعها الثابت لدينا هي 1.43 مرة أسرع في فك التشفير، مع الحفاظ على جودة الترجمة على مجموعة واسعة من مهام الترجمة الآلية عند استخدامها في شبكات فك تشفير الذات والانتباه.
The Transformer translation model is based on the multi-head attention mechanism, which can be parallelized easily. The multi-head attention network performs the scaled dot-product attention function in parallel, empowering the model by jointly attending to information from different representation subspaces at different positions. In this paper, we present an approach to learning a hard retrieval attention where an attention head only attends to one token in the sentence rather than all tokens. The matrix multiplication between attention probabilities and the value sequence in the standard scaled dot-product attention can thus be replaced by a simple and efficient retrieval operation. We show that our hard retrieval attention mechanism is 1.43 times faster in decoding, while preserving translation quality on a wide range of machine translation tasks when used in the decoder self- and cross-attention networks.
References used
https://aclanthology.org/
rate research
Read More
Self-supervised learning has recently attracted considerable attention in the NLP community for its ability to learn discriminative features using a contrastive objective. This paper investigates whether contrastive learning can be extended to Transf
omer attention to tackling the Winograd Schema Challenge. To this end, we propose a novel self-supervised framework, leveraging a contrastive loss directly at the level of self-attention. Experimental analysis of our attention-based models on multiple datasets demonstrates superior commonsense reasoning capabilities. The proposed approach outperforms all comparable unsupervised approaches while occasionally surpassing supervised ones.
Ever since neural models were adopted in data-to-text language generation, they have invariably been reliant on extrinsic components to improve their semantic accuracy, because the models normally do not exhibit the ability to generate text that reli
ably mentions all of the information provided in the input. In this paper, we propose a novel decoding method that extracts interpretable information from encoder-decoder models' cross-attention, and uses it to infer which attributes are mentioned in the generated text, which is subsequently used to rescore beam hypotheses. Using this decoding method with T5 and BART, we show on three datasets its ability to dramatically reduce semantic errors in the generated outputs, while maintaining their state-of-the-art quality.
In this work, we conduct a comprehensive investigation on one of the centerpieces of modern machine translation systems: the encoder-decoder attention mechanism. Motivated by the concept of first-order alignments, we extend the (cross-)attention mech
anism by a recurrent connection, allowing direct access to previous attention/alignment decisions. We propose several ways to include such a recurrency into the attention mechanism. Verifying their performance across different translation tasks we conclude that these extensions and dependencies are not beneficial for the translation performance of the Transformer architecture.
Natural question generation (QG) aims to generate questions from a passage, and generated questions are answered from the passage. Most models with state-of-the-art performance model the previously generated text at each decoding step. However, (1) t
hey ignore the rich structure information that is hidden in the previously generated text. (2) they ignore the impact of copied words on the passage. We perceive that information in previously generated words serves as auxiliary information in subsequent generation. To address these problems, we design the Iterative Graph Network-based Decoder (IGND) to model the previous generation using a Graph Neural Network at each decoding step. Moreover, our graph model captures dependency relations in the passage that boost the generation. Experimental results demonstrate that our model outperforms the state-of-the-art models with sentence-level QG tasks on SQuAD and MARCO datasets.
Transformer models are permutation equivariant. To supply the order and type information of the input tokens, position and segment embeddings are usually added to the input. Recent works proposed variations of positional encodings with relative posit
ion encodings achieving better performance. Our analysis shows that the gain actually comes from moving positional information to attention layer from the input. Motivated by this, we introduce Decoupled Positional Attention for Transformers (DIET), a simple yet effective mechanism to encode position and segment information into the Transformer models. The proposed method has faster training and inference time, while achieving competitive performance on GLUE, XTREME and WMT benchmarks. We further generalize our method to long-range transformers and show performance gain.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
