Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userTowards Improving Adversarial Training of NLP Models
نحو تحسين التدريب الخصم لنماذج NLP
179
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تدريب الخصم، طريقة لتعلم الشبكات العصبية العميقة القوية، تضم أمثلة خصومة أثناء التدريب. ومع ذلك، فإن الأساليب الحديثة لتوليد أمثلة allp adversarial تنطوي على البحث عن الفروضي وترميز الجملة باهظة الثمن لتقييد الحالات التي تم إنشاؤها. نتيجة لذلك، لا يزال يمثل تحديا لاستخدام التدريب المشدود الفانيليا لتحسين أداء نماذج NLP، والفوائد غير مرئية بشكل أساسي. تقترح هذه الورقة عملية تدريبية بسيطة ومحسنة من الفانيليا العدائية لنماذج NLP، والتي نستها المهاجمة على التدريب (A2T). الجزء الأساسي من A2T هو هجوم استبدال كلمة جديدة وأرخص محسن لتدريب الفانيليا الخصم. نحن نستخدم A2T لتدريب برت ونماذج روبرتا على مجموعة بيانات IMDB والطماطم الفاسدة والشبكة الصلبة و SNLI. تظهر نتائجنا تجريبيا أنه من الممكن تدريب نماذج NLP قوية باستخدام خصم أرخص بكثير. نوضح أن التدريب الصادق للفانيليا مع A2T يمكن أن يحسن متانة نموذج NLP للهجوم الذي تم تدريبه في الأصل مع النموذج الذي يدافع عنه أيضا ضد أنواع أخرى من هجمات استبدال الكلمات. علاوة على ذلك، نظهر أن A2T يمكن أن تحسن الدقة القياسية لنماذج NLP وتعميم المجال المتبادل والتفسيرية.
Adversarial training, a method for learning robust deep neural networks, constructs adversarial examples during training. However, recent methods for generating NLP adversarial examples involve combinatorial search and expensive sentence encoders for constraining the generated instances. As a result, it remains challenging to use vanilla adversarial training to improve NLP models' performance, and the benefits are mainly uninvestigated. This paper proposes a simple and improved vanilla adversarial training process for NLP models, which we name Attacking to Training (A2T). The core part of A2T is a new and cheaper word substitution attack optimized for vanilla adversarial training. We use A2T to train BERT and RoBERTa models on IMDB, Rotten Tomatoes, Yelp, and SNLI datasets. Our results empirically show that it is possible to train robust NLP models using a much cheaper adversary. We demonstrate that vanilla adversarial training with A2T can improve an NLP model's robustness to the attack it was originally trained with and also defend the model against other types of word substitution attacks. Furthermore, we show that A2T can improve NLP models' standard accuracy, cross-domain generalization, and interpretability.
References used
https://aclanthology.org/
rate research
Read More
We present a simple yet effective Targeted Adversarial Training (TAT) algorithm to improve adversarial training for natural language understanding. The key idea is to introspect current mistakes and prioritize adversarial training steps to where the
model errs the most. Experiments show that TAT can significantly improve accuracy over standard adversarial training on GLUE and attain new state-of-the-art zero-shot results on XNLI. Our code will be released upon acceptance of the paper.
Adversarial training (AT) as a regularization method has proved its effectiveness on various tasks. Though there are successful applications of AT on some NLP tasks, the distinguishing characteristics of NLP tasks have not been exploited. In this pap
er, we aim to apply AT on machine reading comprehension (MRC) tasks. Furthermore, we adapt AT for MRC tasks by proposing a novel adversarial training method called PQAT that perturbs the embedding matrix instead of word vectors. To differentiate the roles of passages and questions, PQAT uses additional virtual P/Q-embedding matrices to gather the global perturbations of words from passages and questions separately. We test the method on a wide range of MRC tasks, including span-based extractive RC and multiple-choice RC. The results show that adversarial training is effective universally, and PQAT further improves the performance.
Abstract Debugging a machine learning model is hard since the bug usually involves the training data and the learning process. This becomes even harder for an opaque deep learning model if we have no clue about how the model actually works. In this s
urvey, we review papers that exploit explanations to enable humans to give feedback and debug NLP models. We call this problem explanation-based human debugging (EBHD). In particular, we categorize and discuss existing work along three dimensions of EBHD (the bug context, the workflow, and the experimental setting), compile findings on how EBHD components affect the feedback providers, and highlight open problems that could be future research directions.
Recently, pre-trained language models (LMs) have achieved strong performance when fine-tuned on difficult benchmarks like SuperGLUE. However, performance can suffer when there are very few labeled examples available for fine-tuning. Pattern Exploitin
g Training (PET) is a recent approach that leverages patterns for few-shot learning. However, PET uses task-specific unlabeled data. In this paper, we focus on few-shot learning without any unlabeled data and introduce ADAPET, which modifies PET's objective to provide denser supervision during fine-tuning. As a result, ADAPET outperforms PET on SuperGLUE without any task-specific unlabeled data.
Cross-target generalization constitutes an important issue for news Stance Detection (SD). In this short paper, we investigate adversarial cross-genre SD, where knowledge from annotated user-generated data is leveraged to improve news SD on targets u
nseen during training. We implement a BERT-based adversarial network and show experimental performance improvements over a set of strong baselines. Given the abundance of user-generated data, which are considerably less expensive to retrieve and annotate than news articles, this constitutes a promising research direction.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
