Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userContrastive Out-of-Distribution Detection for Pretrained Transformers
كشف خارج التوزيع من غير مسبوق للمحولات المحددة
438
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
يحقق المحولات مسبقا أداء ملحوظا عند التدريب وبيانات الاختبار من نفس التوزيع. ومع ذلك، في سيناريوهات العالم الحقيقي، غالبا ما يواجه النموذج حالات خارج التوزيع (OOD) التي يمكن أن تسبب مشاكل شديدة التحول الدلالي في وقت الاستدلال. لذلك، في الممارسة العملية، يجب على نموذج موثوق أن يحدد هذه الحالات، ثم رفضها أثناء الاستدلال أو نقلها إلى النماذج التي تتعامل مع توزيع آخر. في هذه الورقة، نقوم بتطوير طريقة اكتشاف OOD غير مزودة بها، حيث يتم استخدام البيانات الموجودة في التوزيع فقط في التدريب. نقترح أن يلزم المحولات بفقدان مقنعين، مما يحسن من إيصال التمثيلات، بحيث يمكن التمييز بين مثيلات OOD بشكل أفضل عن المعرف. يمكن بعد ذلك اكتشاف هذه الحالات OOD بدقة باستخدام مسافة Mahalanobis في الطبقة السابقة للاشمئزاز. نقوم بتجربة إعدادات شاملة وتحقيق أداء الكشف عن المسؤولية المثالية تقريبا، وتفوق خطوط الأساس بشكل كبير. نحن مزيد من التحقيق في المناولة وراء التحسن، مما يجد أن المزيد من التمثيلات المدمجة من خلال التعلم المتعاقل الذي يعتمد على الهامش يجلب التحسن. نطلق سرد علاماتنا للمجتمع للبحث في المستقبل.
Pretrained Transformers achieve remarkable performance when training and test data are from the same distribution. However, in real-world scenarios, the model often faces out-of-distribution (OOD) instances that can cause severe semantic shift problems at inference time. Therefore, in practice, a reliable model should identify such instances, and then either reject them during inference or pass them over to models that handle another distribution. In this paper, we develop an unsupervised OOD detection method, in which only the in-distribution (ID) data are used in training. We propose to fine-tune the Transformers with a contrastive loss, which improves the compactness of representations, such that OOD instances can be better differentiated from ID ones. These OOD instances can then be accurately detected using the Mahalanobis distance in the model's penultimate layer. We experiment with comprehensive settings and achieve near-perfect OOD detection performance, outperforming baselines drastically. We further investigate the rationales behind the improvement, finding that more compact representations through margin-based contrastive learning bring the improvement. We release our code to the community for future research.
References used
https://aclanthology.org/
rate research
Read More
While neural networks are ubiquitous in state-of-the-art semantic parsers, it has been shown that most standard models suffer from dramatic performance losses when faced with compositionally out-of-distribution (OOD) data. Recently several methods ha
ve been proposed to improve compositional generalization in semantic parsing. In this work we instead focus on the problem of detecting compositionally OOD examples with neural semantic parsers, which, to the best of our knowledge, has not been investigated before. We investigate several strong yet simple methods for OOD detection based on predictive uncertainty. The experimental results demonstrate that these techniques perform well on the standard SCAN and CFQ datasets. Moreover, we show that OOD detection can be further improved by using a heterogeneous ensemble.
After a neural sequence model encounters an unexpected token, can its behavior be predicted? We show that RNN and transformer language models exhibit structured, consistent generalization in out-of-distribution contexts. We begin by introducing two i
dealized models of generalization in next-word prediction: a lexical context model in which generalization is consistent with the last word observed, and a syntactic context model in which generalization is consistent with the global structure of the input. In experiments in English, Finnish, Mandarin, and random regular languages, we demonstrate that neural language models interpolate between these two forms of generalization: their predictions are well-approximated by a log-linear combination of lexical and syntactic predictive distributions. We then show that, in some languages, noise mediates the two forms of generalization: noise applied to input tokens encourages syntactic generalization, while noise in history representations encourages lexical generalization. Finally, we offer a preliminary theoretical explanation of these results by proving that the observed interpolation behavior is expected in log-linear models with a particular feature correlation structure. These results help explain the effectiveness of two popular regularization schemes and show that aspects of sequence model generalization can be understood and controlled.
We describe our system that ranked first in Hope Speech Detection (HSD) shared task and fourth in Offensive Language Identification (OLI) shared task, both in Tamil language. The goal of HSD and OLI is to identify if a code-mixed comment or post cont
ains hope speech or offensive content respectively. We pre-train a transformer-based model RoBERTa using synthetically generated code-mixed data and use it in an ensemble along with their pre-trained ULMFiT model available from iNLTK.
Detecting out-of-domain (OOD) intents is crucial for the deployed task-oriented dialogue system. Previous unsupervised OOD detection methods only extract discriminative features of different in-domain intents while supervised counterparts can directl
y distinguish OOD and in-domain intents but require extensive labeled OOD data. To combine the benefits of both types, we propose a self-supervised contrastive learning framework to model discriminative semantic features of both in-domain intents and OOD intents from unlabeled data. Besides, we introduce an adversarial augmentation neural module to improve the efficiency and robustness of contrastive learning. Experiments on two public benchmark datasets show that our method can consistently outperform the baselines with a statistically significant margin.
Successful methods for unsupervised neural machine translation (UNMT) employ cross-lingual pretraining via self-supervision, often in the form of a masked language modeling or a sequence generation task, which requires the model to align the lexical-
and high-level representations of the two languages. While cross-lingual pretraining works for similar languages with abundant corpora, it performs poorly in low-resource and distant languages. Previous research has shown that this is because the representations are not sufficiently aligned. In this paper, we enhance the bilingual masked language model pretraining with lexical-level information by using type-level cross-lingual subword embeddings. Empirical results demonstrate improved performance both on UNMT (up to 4.5 BLEU) and bilingual lexicon induction using our method compared to a UNMT baseline.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
