Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userUniversal-KD: Attention-based Output-Grounded Intermediate Layer Knowledge Distillation
عالمي د.ك: تقطير المعرفة الطبقات المتوسطة القائمة على الانتباه
257
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
يظهر مطابقة الطبقة الوسيطة كهدوث فعال لتحسين تقطير المعرفة (KD). ومع ذلك، تنطبق هذه التقنية مطابقة في المساحات المخفية لشبكتين مختلفتين (أي طالب ومدرس)، والتي تفتقر إلى التفسير الواضح. علاوة على ذلك، لا يمكن للطبقة المتوسطة KD التعامل بسهولة مع مشاكل أخرى مثل البحث عن تعيين الطبقة وعدم عدم تطابق الهندسة المعمارية (أي أن المعلم والطالب ليكون من نفس النوع النموذجي). لمعالجة المشاكل المذكورة أعلاه، نقترح عالمي دينار كويتي لمطابقة الطبقات الوسيطة من المعلم والطالب في مساحة الإخراج (عن طريق إضافة مصنفات زائفة على الطبقات المتوسطة) عبر إسقاط الطبقة المستندة إلى الاهتمام. من خلال القيام بذلك، يتمتع نهجنا الموحد بثلاث مزايا: (1) يمكن دمجها بمرونة مع تقنيات تقطير الطبقة المتوسطة الحالية لتحسين نتائجها (2) يمكن نشر مصنفات الزائفة من المعلم بدلا من شبكات مساعد المعلم باهظة الثمن مشكلة فجوة القدرة في KD وهي مشكلة شائعة عندما تصبح الفجوة بين حجم المعلم وشبكات الطلاب كبيرة جدا؛ (3) يمكن استخدامه في الطبقة الوسيطة عبر الهندسة الوسطى دينار كويتي. لقد قمنا بتجارب شاملة في تقطير Bert-Base في Bert-4، Roberta-Large في Distilroberta وقاعدة Bert-Base في نماذج CNN و LSTM. تظهر النتائج على مهام الغراء أن نهجنا قادر على تفوق تقنيات KD الأخرى.
Intermediate layer matching is shown as an effective approach for improving knowledge distillation (KD). However, this technique applies matching in the hidden spaces of two different networks (i.e. student and teacher), which lacks clear interpretability. Moreover, intermediate layer KD cannot easily deal with other problems such as layer mapping search and architecture mismatch (i.e. it requires the teacher and student to be of the same model type). To tackle the aforementioned problems all together, we propose Universal-KD to match intermediate layers of the teacher and the student in the output space (by adding pseudo classifiers on intermediate layers) via the attention-based layer projection. By doing this, our unified approach has three merits: (i) it can be flexibly combined with current intermediate layer distillation techniques to improve their results (ii) the pseudo classifiers of the teacher can be deployed instead of extra expensive teacher assistant networks to address the capacity gap problem in KD which is a common issue when the gap between the size of the teacher and student networks becomes too large; (iii) it can be used in cross-architecture intermediate layer KD. We did comprehensive experiments in distilling BERT-base into BERT-4, RoBERTa-large into DistilRoBERTa and BERT-base into CNN and LSTM-based models. Results on the GLUE tasks show that our approach is able to outperform other KD techniques.
References used
https://aclanthology.org/
rate research
Read More
Recent studies argue that knowledge distillation is promising for speech translation (ST) using end-to-end models. In this work, we investigate the effect of knowledge distillation with a cascade ST using automatic speech recognition (ASR) and machin
e translation (MT) models. We distill knowledge from a teacher model based on human transcripts to a student model based on erroneous transcriptions. Our experimental results demonstrated that knowledge distillation is beneficial for a cascade ST. Further investigation that combined knowledge distillation and fine-tuning revealed that the combination consistently improved two language pairs: English-Italian and Spanish-English.
To reduce a model size but retain performance, we often rely on knowledge distillation (KD) which transfers knowledge from a large teacher'' model to a smaller student'' model. However, KD on multimodal datasets such as vision-language tasks is relat
ively unexplored, and digesting multimodal information is challenging since different modalities present different types of information. In this paper, we perform a large-scale empirical study to investigate the importance and effects of each modality in knowledge distillation. Furthermore, we introduce a multimodal knowledge distillation framework, modality-specific distillation (MSD), to transfer knowledge from a teacher on multimodal tasks by learning the teacher's behavior within each modality. The idea aims at mimicking a teacher's modality-specific predictions by introducing auxiliary loss terms for each modality. Furthermore, because each modality has different saliency for predictions, we define saliency scores for each modality and investigate saliency-based weighting schemes for the auxiliary losses. We further study a weight learning approach to learn the optimal weights on these loss terms. In our empirical analysis, we examine the saliency of each modality in KD, demonstrate the effectiveness of the weighting scheme in MSD, and show that it achieves better performance than KD on four multimodal datasets.
Although pre-trained big models (e.g., BERT, ERNIE, XLNet, GPT3 etc.) have delivered top performance in Seq2seq modeling, their deployments in real-world applications are often hindered by the excessive computations and memory demand involved. For ma
ny applications, including named entity recognition (NER), matching the state-of-the-art result under budget has attracted considerable attention. Drawing power from the recent advance in knowledge distillation (KD), this work presents a novel distillation scheme to efficiently transfer the knowledge learned from big models to their more affordable counterpart. Our solution highlights the construction of surrogate labels through the k-best Viterbi algorithm to distill knowledge from the teacher model. To maximally assimilate knowledge into the student model, we propose a multi-grained distillation scheme, which integrates cross entropy involved in conditional random field (CRF) and fuzzy learning.To validate the effectiveness of our proposal, we conducted a comprehensive evaluation on five NER benchmarks, reporting cross-the-board performance gains relative to competing prior-arts. We further discuss ablation results to dissect our gains.
In this paper we apply self-knowledge distillation to text summarization which we argue can alleviate problems with maximum-likelihood training on single reference and noisy datasets. Instead of relying on one-hot annotation labels, our student summa
rization model is trained with guidance from a teacher which generates smoothed labels to help regularize training. Furthermore, to better model uncertainty during training, we introduce multiple noise signals for both teacher and student models. We demonstrate experimentally on three benchmarks that our framework boosts the performance of both pretrained and non-pretrained summarizers achieving state-of-the-art results.
Pretrained transformer-based encoders such as BERT have been demonstrated to achieve state-of-the-art performance on numerous NLP tasks. Despite their success, BERT style encoders are large in size and have high latency during inference (especially o
n CPU machines) which make them unappealing for many online applications. Recently introduced compression and distillation methods have provided effective ways to alleviate this shortcoming. However, the focus of these works has been mainly on monolingual encoders. Motivated by recent successes in zero-shot cross-lingual transfer learning using multilingual pretrained encoders such as mBERT, we evaluate the effectiveness of Knowledge Distillation (KD) both during pretraining stage and during fine-tuning stage on multilingual BERT models. We demonstrate that in contradiction to the previous observation in the case of monolingual distillation, in multilingual settings, distillation during pretraining is more effective than distillation during fine-tuning for zero-shot transfer learning. Moreover, we observe that distillation during fine-tuning may hurt zero-shot cross-lingual performance. Finally, we demonstrate that distilling a larger model (BERT Large) results in the strongest distilled model that performs best both on the source language as well as target languages in zero-shot settings.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
