Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userModality-specific Distillation
التقطير الخاص بالطريقة
52
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
الشبكات العصبية الكبيرة غير عملي للنشر على الأجهزة المحمولة بسبب التكلفة الحاسوبية الثقيلة والاستدلال البطيء. تقطير المعرفة (KD) هي تقنية لتقليل حجم النموذج مع الاحتفاظ بالأداء عن طريق نقل المعرفة من نموذج المعلم الكبير إلى نموذج طالب أصغر ". ومع ذلك، فإن KD على مجموعات بيانات متعددة الوسائط مثل مجموعات بيانات اللغة الرؤية غير مستكشفة نسبيا وهضم هذه المعلومات متعددة الوسائط تحديا لأن طرائق مختلفة تقدم أنواعا مختلفة من المعلومات. في هذه الورقة، نقترح تقطير نوعية (MSD) إلى نقل المعرفة بفعالية من المعلم على مجموعات البيانات متعددة الوسائط. يمكن تطبيق نهج KD الحالية على إعداد متعدد الوسائط، لكن الطالب ليس لديه حق الوصول إلى التنبؤات الخاصة بالطريقة. تهدف فكرتنا إلى تحية التنبؤات الخاصة بنوية المعلم من خلال إدخال مصطلح خسائر مساعدة لكل طريقة. نظرا لأن كل طريقة لها أهمية مختلفة للتنبؤات، فإننا نقترح أيضا أساليب التردد للخسائر الإضافية؛ نهج التعلم التلوي لتعلم الأوزان المثلى على شروط الخسارة هذه. في تجاربنا، نوضح فعالية MSD لدينا ونظام الترجيح وإظهار أنه يحقق أداء أفضل من KD.
Large neural networks are impractical to deploy on mobile devices due to their heavy computational cost and slow inference. Knowledge distillation (KD) is a technique to reduce the model size while retaining performance by transferring knowledge from a large teacher'' model to a smaller student'' model. However, KD on multimodal datasets such as vision-language datasets is relatively unexplored and digesting such multimodal information is challenging since different modalities present different types of information. In this paper, we propose modality-specific distillation (MSD) to effectively transfer knowledge from a teacher on multimodal datasets. Existing KD approaches can be applied to multimodal setup, but a student doesn't have access to modality-specific predictions. Our idea aims at mimicking a teacher's modality-specific predictions by introducing an auxiliary loss term for each modality. Because each modality has different importance for predictions, we also propose weighting approaches for the auxiliary losses; a meta-learning approach to learn the optimal weights on these loss terms. In our experiments, we demonstrate the effectiveness of our MSD and the weighting scheme and show that it achieves better performance than KD.
References used
https://aclanthology.org/
rate research
Read More
In most of neural machine translation distillation or stealing scenarios, the highest-scoring hypothesis of the target model (teacher) is used to train a new model (student). If reference translations are also available, then better hypotheses (with
respect to the references) can be oversampled and poor hypotheses either removed or undersampled. This paper explores the sampling method landscape (pruning, hypothesis oversampling and undersampling, deduplication and their combination) with English to Czech and English to German MT models using standard MT evaluation metrics. We show that careful oversampling and combination with the original data leads to better performance when compared to training only on the original or synthesized data or their direct combination.
Recent advances in NLP systems, notably the pretraining-and-finetuning paradigm, have achieved great success in predictive accuracy. However, these systems are usually not well calibrated for uncertainty out-of-the-box. Many recalibration methods hav
e been proposed in the literature for quantifying predictive uncertainty and calibrating model outputs, with varying degrees of complexity. In this work, we present a systematic study of a few of these methods. Focusing on the text classification task and finetuned large pretrained language models, we first show that many of the finetuned models are not well calibrated out-of-the-box, especially when the data come from out-of-domain settings. Next, we compare the effectiveness of a few widely-used recalibration methods (such as ensembles, temperature scaling). Then, we empirically illustrate a connection between distillation and calibration. We view distillation as a regularization term encouraging the student model to output uncertainties that match those of a teacher model. With this insight, we develop simple recalibration methods based on distillation with no additional inference-time cost. We show on the GLUE benchmark that our simple methods can achieve competitive out-of-domain (OOD) calibration performance w.r.t. more expensive approaches. Finally, we include ablations to understand the usefulness of components of our proposed method and examine the transferability of calibration via distillation.
We present an algorithm based on multi-layer transformers for identifying Adverse Drug Reactions (ADR) in social media data. Our model relies on the properties of the problem and the characteristics of contextual word embeddings to extract two views
from documents. Then a classifier is trained on each view to label a set of unlabeled documents to be used as an initializer for a new classifier in the other view. Finally, the initialized classifier in each view is further trained using the initial training examples. We evaluated our model in the largest publicly available ADR dataset. The experiments testify that our model significantly outperforms the transformer-based models pretrained on domain-specific data.
Understanding linguistic modality is widely seen as important for downstream tasks such as Question Answering and Knowledge Graph Population. Entailment Graph learning might also be expected to benefit from attention to modality. We build Entailment
Graphs using a news corpus filtered with a modality parser, and show that stripping modal modifiers from predicates in fact increases performance. This suggests that for some tasks, the pragmatics of modal modification of predicates allows them to contribute as evidence of entailment.
Spoken question answering (SQA) requires fine-grained understanding of both spoken documents and questions for the optimal answer prediction. In this paper, we propose novel training schemes for spoken question answering with a self-supervised traini
ng stage and a contrastive representation learning stage. In the self-supervised stage, we propose three auxiliary self-supervised tasks, including utterance restoration, utterance insertion, and question discrimination, and jointly train the model to capture consistency and coherence among speech documents without any additional data or annotations. We then propose to learn noise-invariant utterance representations in a contrastive objective by adopting multiple augmentation strategies, including span deletion and span substitution. Besides, we design a Temporal-Alignment attention to semantically align the speech-text clues in the learned common space and benefit the SQA tasks. By this means, the training schemes can more effectively guide the generation model to predict more proper answers. Experimental results show that our model achieves state-of-the-art results on three SQA benchmarks. Our code will be publicly available after publication.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
