Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userLimitations of Knowledge Distillation for Zero-shot Transfer Learning
حدود تقطير المعرفة لتعلم تحويل الصفر
318
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تم إثبات المشفرات المستندة إلى المحولات المسبدة مسبقا مثل بيرت لتحقيق الأداء الحديث في العديد من مهام NLP العديدة. على الرغم من نجاحهم، فإن ترميز نمط بيرت كبير الحجم ولديها زمن بيانات عالية أثناء الاستدلال (خاصة في آلات وحدة المعالجة المركزية) مما يجعلها غير جذابة للعديد من التطبيقات عبر الإنترنت. قدمت أساليب الضغط والتقطير مؤخرا طرقا فعالة لتخفيف هذا القصور. ومع ذلك، فإن محور هذه الأعمال كان أساسا في ترميز أحادي الأونلينغ. بدافع من النجاحات الأخيرة في التعلم عبر التحويل المتبادل في صفر تسديدة باستخدام ترميز مسببات اللغات المسبق، مثل MBERT، فإننا نقيم فعالية تقطير المعرفة (دينار كويتي) خلال مرحلة الاحتجاج وأثناء مرحلة ضبط الدقيقة على نماذج بيرت متعددة اللغات. نوضح أنه في تناقض الملاحظة السابقة في حالة التقطير أحادي الأونلينغ، في الإعدادات المتعددة اللغات، يكون التقطير أثناء الاحتجاز أكثر فعالية من التقطير أثناء ضبط الصفر عن التعلم تحويل الصفر. علاوة على ذلك، فإننا نلاحظ أن التقطير أثناء ضبط الرصيف قد يضر أداء الصفر اللغوي الصفر. أخيرا، نوضح أن تقطير نموذج أكبر (بيرت كبير) ينتج عن أقوى النموذج المقطر الذي يؤدي أفضل سواء على لغة المصدر وكذلك اللغات المستهدفة في إعدادات الطلقة الصفرية.
Pretrained transformer-based encoders such as BERT have been demonstrated to achieve state-of-the-art performance on numerous NLP tasks. Despite their success, BERT style encoders are large in size and have high latency during inference (especially on CPU machines) which make them unappealing for many online applications. Recently introduced compression and distillation methods have provided effective ways to alleviate this shortcoming. However, the focus of these works has been mainly on monolingual encoders. Motivated by recent successes in zero-shot cross-lingual transfer learning using multilingual pretrained encoders such as mBERT, we evaluate the effectiveness of Knowledge Distillation (KD) both during pretraining stage and during fine-tuning stage on multilingual BERT models. We demonstrate that in contradiction to the previous observation in the case of monolingual distillation, in multilingual settings, distillation during pretraining is more effective than distillation during fine-tuning for zero-shot transfer learning. Moreover, we observe that distillation during fine-tuning may hurt zero-shot cross-lingual performance. Finally, we demonstrate that distilling a larger model (BERT Large) results in the strongest distilled model that performs best both on the source language as well as target languages in zero-shot settings.
References used
https://aclanthology.org/
rate research
Read More
To reduce a model size but retain performance, we often rely on knowledge distillation (KD) which transfers knowledge from a large teacher'' model to a smaller student'' model. However, KD on multimodal datasets such as vision-language tasks is relat
ively unexplored, and digesting multimodal information is challenging since different modalities present different types of information. In this paper, we perform a large-scale empirical study to investigate the importance and effects of each modality in knowledge distillation. Furthermore, we introduce a multimodal knowledge distillation framework, modality-specific distillation (MSD), to transfer knowledge from a teacher on multimodal tasks by learning the teacher's behavior within each modality. The idea aims at mimicking a teacher's modality-specific predictions by introducing auxiliary loss terms for each modality. Furthermore, because each modality has different saliency for predictions, we define saliency scores for each modality and investigate saliency-based weighting schemes for the auxiliary losses. We further study a weight learning approach to learn the optimal weights on these loss terms. In our empirical analysis, we examine the saliency of each modality in KD, demonstrate the effectiveness of the weighting scheme in MSD, and show that it achieves better performance than KD on four multimodal datasets.
In this paper we apply self-knowledge distillation to text summarization which we argue can alleviate problems with maximum-likelihood training on single reference and noisy datasets. Instead of relying on one-hot annotation labels, our student summa
rization model is trained with guidance from a teacher which generates smoothed labels to help regularize training. Furthermore, to better model uncertainty during training, we introduce multiple noise signals for both teacher and student models. We demonstrate experimentally on three benchmarks that our framework boosts the performance of both pretrained and non-pretrained summarizers achieving state-of-the-art results.
Multilingual pre-trained contextual embedding models (Devlin et al., 2019) have achieved impressive performance on zero-shot cross-lingual transfer tasks. Finding the most effective fine-tuning strategy to fine-tune these models on high-resource lang
uages so that it transfers well to the zero-shot languages is a non-trivial task. In this paper, we propose a novel meta-optimizer to soft-select which layers of the pre-trained model to freeze during fine-tuning. We train the meta-optimizer by simulating the zero-shot transfer scenario. Results on cross-lingual natural language inference show that our approach improves over the simple fine-tuning baseline and X-MAML (Nooralahzadeh et al., 2020).
Recent studies argue that knowledge distillation is promising for speech translation (ST) using end-to-end models. In this work, we investigate the effect of knowledge distillation with a cascade ST using automatic speech recognition (ASR) and machin
e translation (MT) models. We distill knowledge from a teacher model based on human transcripts to a student model based on erroneous transcriptions. Our experimental results demonstrated that knowledge distillation is beneficial for a cascade ST. Further investigation that combined knowledge distillation and fine-tuning revealed that the combination consistently improved two language pairs: English-Italian and Spanish-English.
Multilingual question answering over knowledge graph (KGQA) aims to derive answers from a knowledge graph (KG) for questions in multiple languages. To be widely applicable, we focus on its zero-shot transfer setting. That is, we can only access train
ing data in a high-resource language, while need to answer multilingual questions without any labeled data in target languages. A straightforward approach is resorting to pre-trained multilingual models (e.g., mBERT) for cross-lingual transfer, but there is a still significant gap of KGQA performance between source and target languages. In this paper, we exploit unsupervised bilingual lexicon induction (BLI) to map training questions in source language into those in target language as augmented training data, which circumvents language inconsistency between training and inference. Furthermore, we propose an adversarial learning strategy to alleviate syntax-disorder of the augmented data, making the model incline to both language- and syntax-independence. Consequently, our model narrows the gap in zero-shot cross-lingual transfer. Experiments on two multilingual KGQA datasets with 11 zero-resource languages verify its effectiveness.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
