Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userInstance-adaptive training with noise-robust losses against noisy labels
على سبيل المثال - التدريب التكيفي مع خسائر قوية ضد الضوضاء ضد الملصقات الصاخبة
453
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
من أجل تخفيف الطلب الكبير على مجموعات البيانات المشروح للمهام المختلفة، اعتمدت العديد من مجموعات بيانات معالجة اللغات الطبيعية الحديثة خطوط أنابيب آلية للبيانات القابلة للاستخدام السريع. ومع ذلك، فإن التدريب النموذجي مع مثل هذه البيانات يشكل تحديا لأن أهداف التحسين الشائعة ليست قوية لتسمية الضوضاء الناجمة عن عملية توليد التوضيحية. تم اقتراح العديد من الخسائر القوية للضوضاء وتقييمها في المهام في رؤية الكمبيوتر، لكنها تستخدم عموما فرط DataSet-WiseParamter واحد للتحكم في قوة مقاومة الضوضاء. يقترح هذا العمل أطر تدريبية جديدة على سبيل المثال لتغيير فرط بيانات DataSet واحد من مقاومة الضوضاء في مثل هذه الخسائر لتكون مثالا. توقع هؤلاء مثيل - Hyperparameters مقاومة للضوضاء من خلال تنبؤات ذات جودة عالية على مستوى التصنيف، والتي يتم تدريبها مع نماذج التصنيف الرئيسية. تظهر تجارب مجموعات بيانات NLP الصاخبة والفساد أن أطر التدريب على سبيل المثال المقترحة على سبيل المثال تساعد في زيادة متانة الضوضاء التي توفرها هذه الخسائر، وتعزيز استخدام الأطر والأطر الخسائر المرتبطة بها في نماذج NLP المدربة مع بيانات صاخبة.
In order to alleviate the huge demand for annotated datasets for different tasks, many recent natural language processing datasets have adopted automated pipelines for fast-tracking usable data. However, model training with such datasets poses a challenge because popular optimization objectives are not robust to label noise induced in the annotation generation process. Several noise-robust losses have been proposed and evaluated on tasks in computer vision, but they generally use a single dataset-wise hyperparamter to control the strength of noise resistance. This work proposes novel instance-adaptive training frameworks to change single dataset-wise hyperparameters of noise resistance in such losses to be instance-wise. Such instance-wise noise resistance hyperparameters are predicted by special instance-level label quality predictors, which are trained along with the main classification models. Experiments on noisy and corrupted NLP datasets show that proposed instance-adaptive training frameworks help increase the noise-robustness provided by such losses, promoting the use of the frameworks and associated losses in NLP models trained with noisy data.
References used
https://aclanthology.org/
rate research
Read More
We present new state-of-the-art benchmarks for paraphrase detection on all six languages in the Opusparcus sentential paraphrase corpus: English, Finnish, French, German, Russian, and Swedish. We reach these baselines by fine-tuning BERT. The best re
sults are achieved on smaller and cleaner subsets of the training sets than was observed in previous research. Additionally, we study a translation-based approach that is competitive for the languages with more limited and noisier training data.
Recent information extraction approaches have relied on training deep neural models. However, such models can easily overfit noisy labels and suffer from performance degradation. While it is very costly to filter noisy labels in large learning resour
ces, recent studies show that such labels take more training steps to be memorized and are more frequently forgotten than clean labels, therefore are identifiable in training. Motivated by such properties, we propose a simple co-regularization framework for entity-centric information extraction, which consists of several neural models with identical structures but different parameter initialization. These models are jointly optimized with the task-specific losses and are regularized to generate similar predictions based on an agreement loss, which prevents overfitting on noisy labels. Extensive experiments on two widely used but noisy benchmarks for information extraction, TACRED and CoNLL03, demonstrate the effectiveness of our framework. We release our code to the community for future research.
A private learning scheme TextHide was recently proposed to protect the private text data during the training phase via so-called instance encoding. We propose a novel reconstruction attack to break TextHide by recovering the private training data, a
nd thus unveil the privacy risks of instance encoding. We have experimentally validated the effectiveness of the reconstruction attack with two commonly-used datasets for sentence classification. Our attack would advance the development of privacy preserving machine learning in the context of natural language processing.
A key challenge of dialog systems research is to effectively and efficiently adapt to new domains. A scalable paradigm for adaptation necessitates the development of generalizable models that perform well in few-shot settings. In this paper, we focus
on the intent classification problem which aims to identify user intents given utterances addressed to the dialog system. We propose two approaches for improving the generalizability of utterance classification models: (1) observers and (2) example-driven training. Prior work has shown that BERT-like models tend to attribute a significant amount of attention to the [CLS] token, which we hypothesize results in diluted representations. Observers are tokens that are not attended to, and are an alternative to the [CLS] token as a semantic representation of utterances. Example-driven training learns to classify utterances by comparing to examples, thereby using the underlying encoder as a sentence similarity model. These methods are complementary; improving the representation through observers allows the example-driven model to better measure sentence similarities. When combined, the proposed methods attain state-of-the-art results on three intent prediction datasets (banking77, clinc150, hwu64) in both the full data and few-shot (10 examples per intent) settings. Furthermore, we demonstrate that the proposed approach can transfer to new intents and across datasets without any additional training.
Deep neural networks are vulnerable to adversarial attacks, where a small perturbation to an input alters the model prediction. In many cases, malicious inputs intentionally crafted for one model can fool another model. In this paper, we present the
first study to systematically investigate the transferability of adversarial examples for text classification models and explore how various factors, including network architecture, tokenization scheme, word embedding, and model capacity, affect the transferability of adversarial examples. Based on these studies, we propose a genetic algorithm to find an ensemble of models that can be used to induce adversarial examples to fool almost all existing models. Such adversarial examples reflect the defects of the learning process and the data bias in the training set. Finally, we derive word replacement rules that can be used for model diagnostics from these adversarial examples.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
