Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userProbing Word Translations in the Transformer and Trading Decoder for Encoder Layers
التحقيق ترجمات كلمة في محول ومكتشف التداول لطبقات التشفير
253
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
نظرا لفعاليتها وأدائها، اجتذب نموذج الترجمة المحولات اهتماما واسعا، مؤخرا من حيث النهج القائمة على التحقيق. يركز العمل السابق على استخدام أو التحقيق في الميزات اللغوية المصدر في التشفير. حتى الآن، فإن الطريقة التي تتطور فيها ترجمة كلمة تتطور في طبقات المحولات لم يتم التحقيق فيها بعد. ساذجا، قد يفترض المرء أن طبقات التشفير التقاط معلومات المصدر أثناء ترجمة طبقات فك التشفير. في هذا العمل، نظير على أن هذا ليس كذلك: الترجمة تحدث بالفعل تدريجيا في طبقات التشفير وحتى في تضمين المدخلات. أكثر من المستغرب، نجد أن بعض طبقات وحدة فك التشفير المنخفضة لا تفعل ذلك بالفعل فك التشفير. نعرض كل هذا من حيث النهج التحقيق حيث نعلم تمثيلات الطبقة التي تم تحليلها إلى مستوى التصنيف المدربين والمجمد النهائي من وحدة فك الترميز المحول لقياس دقة ترجمة Word. تحفز النتائج التي توصلنا إليها وشرح تغيير تكوين محول: إذا حدث الترجمة بالفعل في طبقات التشفير، فربما يمكننا زيادة عدد طبقات التشفير، مع تقليل عدد طبقات فك ترميز أو زيادة سرعة فك التشفير، دون خسارة في جودة الترجمة؟ تبين تجاربنا أن هذا هو في الواقع الحالة: يمكننا زيادة السرعة إلى عامل 2.3 مع مكاسب صغيرة في جودة الترجمة، في حين أن تكوين التشفير العميق 18-4 يعزز جودة الترجمة عن طريق +1.42 بلو (EN-DE) بسرعة -1.4 1.4.
Due to its effectiveness and performance, the Transformer translation model has attracted wide attention, most recently in terms of probing-based approaches. Previous work focuses on using or probing source linguistic features in the encoder. To date, the way word translation evolves in Transformer layers has not yet been investigated. Naively, one might assume that encoder layers capture source information while decoder layers translate. In this work, we show that this is not quite the case: translation already happens progressively in encoder layers and even in the input embeddings. More surprisingly, we find that some of the lower decoder layers do not actually do that much decoding. We show all of this in terms of a probing approach where we project representations of the layer analyzed to the final trained and frozen classifier level of the Transformer decoder to measure word translation accuracy. Our findings motivate and explain a Transformer configuration change: if translation already happens in the encoder layers, perhaps we can increase the number of encoder layers, while decreasing the number of decoder layers, boosting decoding speed, without loss in translation quality? Our experiments show that this is indeed the case: we can increase speed by up to a factor 2.3 with small gains in translation quality, while an 18-4 deep encoder configuration boosts translation quality by +1.42 BLEU (En-De) at a speed-up of 1.4.
References used
https://aclanthology.org/
rate research
Read More
Encoder-decoder models have been commonly used for many tasks such as machine translation and response generation. As previous research reported, these models suffer from generating redundant repetition. In this research, we propose a new mechanism f
or encoder-decoder models that estimates the semantic difference of a source sentence before and after being fed into the encoder-decoder model to capture the consistency between two sides. This mechanism helps reduce repeatedly generated tokens for a variety of tasks. Evaluation results on publicly available machine translation and response generation datasets demonstrate the effectiveness of our proposal.
We probe pre-trained transformer language models for bridging inference. We first investigate individual attention heads in BERT and observe that attention heads at higher layers prominently focus on bridging relations in-comparison with the lower an
d middle layers, also, few specific attention heads concentrate consistently on bridging. More importantly, we consider language models as a whole in our second approach where bridging anaphora resolution is formulated as a masked token prediction task (Of-Cloze test). Our formulation produces optimistic results without any fine-tuning, which indicates that pre-trained language models substantially capture bridging inference. Our further investigation shows that the distance between anaphor-antecedent and the context provided to language models play an important role in the inference.
The paper presents four models submitted to Part 2 of the SIGMORPHON 2021 Shared Task 0, which aims at replicating human judgements on the inflection of nonce lexemes. Our goal is to explore the usefulness of combining pre-compiled analogical pattern
s with an encoder-decoder architecture. Two models are designed using such patterns either in the input or the output of the network. Two extra models controlled for the role of raw similarity of nonce inflected forms to existing inflected forms in the same paradigm cell, and the role of the type frequency of analogical patterns. Our strategy is entirely endogenous in the sense that the models appealing solely to the data provided by the SIGMORPHON organisers, without using external resources. Our model 2 ranks second among all submitted systems, suggesting that the inclusion of analogical patterns in the network architecture is useful in mimicking speakers' predictions.
Feed-forward layers constitute two-thirds of a transformer model's parameters, yet their role in the network remains under-explored. We show that feed-forward layers in transformer-based language models operate as key-value memories, where each key c
orrelates with textual patterns in the training examples, and each value induces a distribution over the output vocabulary. Our experiments show that the learned patterns are human-interpretable, and that lower layers tend to capture shallow patterns, while upper layers learn more semantic ones. The values complement the keys' input patterns by inducing output distributions that concentrate probability mass on tokens likely to appear immediately after each pattern, particularly in the upper layers. Finally, we demonstrate that the output of a feed-forward layer is a composition of its memories, which is subsequently refined throughout the model's layers via residual connections to produce the final output distribution.
This shared task system description depicts two neural network architectures submitted to the ProfNER track, among them the winning system that scored highest in the two sub-tasks 7a and 7b. We present in detail the approach, preprocessing steps and
the architectures used to achieve the submitted results, and also provide a GitHub repository to reproduce the scores. The winning system is based on a transformer-based pretrained language model and solves the two sub-tasks simultaneously.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
