Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userMuppet: Massive Multi-task Representations with Pre-Finetuning
Muppet: تمثيلات مهام ضخمة متعددة مع Finetuning مسبقا
167
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
نقترح بشكل مسبق، مرحلة تعليمية واسعة النطاق على نطاق واسع بين نموذج اللغة قبل التدريب والضبط بشكل جيد.ما قبل التأثيرات المسبقة للتعلم متعدد المهام على نطاق واسع (حوالي 50 مجموعة من مجموعات البيانات، أكثر من 4.8 مليون أمثلة كاملة المسمى)، وهي مصممة لتشجيع تعلم التمثيلات التي تعمم بشكل أفضل للعديد من المهام المختلفة.نظظ أننا نتحسن باستمرار بشكل ثابت على تحسين الأداء من أجل التمييز المسبق (مثل روبرتا) ونماذج الجيل (مثل بارت) على مجموعة واسعة من المهام (تنبؤ الحكم، ومنطق العمولة، وما إلى ذلك)، مع تحسين كفاءة العينة بشكل كبير بشكل كبير أثناء غرامة-ضبط.نظهر أيضا أن المهام المتعدد واسع النطاق أمر بالغ الأهمية؛يمكن أن تؤذي ما قبل الاستخدام مسبقا الأداء عند استخدام عدد قليل من المهام حتى نقطة حرجة (عادة أعلى من 15) بعد أن يحسن الأداء خطيا في عدد المهام.
We propose pre-finetuning, an additional large-scale learning stage between language model pre-training and fine-tuning. Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to encourage learning of representations that generalize better to many different tasks. We show that pre-finetuning consistently improves performance for pretrained discriminators (e.g. RoBERTa) and generation models (e.g. BART) on a wide range of tasks (sentence prediction, commonsense reasoning, MRC, etc.), while also significantly improving sample efficiency during fine-tuning. We also show that large-scale multi-tasking is crucial; pre-finetuning can hurt performance when few tasks are used up until a critical point (usually above 15) after which performance improves linearly in the number of tasks.
References used
https://aclanthology.org/
rate research
Read More
Abstract For natural language processing systems, two kinds of evidence support the use of text representations from neural language models pretrained'' on large unannotated corpora: performance on application-inspired benchmarks (Peters et al., 2018
, inter alia), and the emergence of syntactic abstractions in those representations (Tenney et al., 2019, inter alia). On the other hand, the lack of grounded supervision calls into question how well these representations can ever capture meaning (Bender and Koller, 2020). We apply novel probes to recent language models--- specifically focusing on predicate-argument structure as operationalized by semantic dependencies (Ivanova et al., 2012)---and find that, unlike syntax, semantics is not brought to the surface by today's pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning, yielding benefits to natural language understanding (NLU) tasks in the GLUE benchmark. This approach demonstrates the potential for general-purpose (rather than task-specific) linguistic supervision, above and beyond conventional pretraining and finetuning. Several diagnostics help to localize the benefits of our approach.1
Text variational autoencoders (VAEs) are notorious for posterior collapse, a phenomenon where the model's decoder learns to ignore signals from the encoder. Because posterior collapse is known to be exacerbated by expressive decoders, Transformers ha
ve seen limited adoption as components of text VAEs. Existing studies that incorporate Transformers into text VAEs (Li et al., 2020; Fang et al., 2021) mitigate posterior collapse using massive pretraining, a technique unavailable to most of the research community without extensive computing resources. We present a simple two-phase training scheme to convert a sequence-to-sequence Transformer into a VAE with just finetuning. The resulting language model is competitive with massively pretrained Transformer-based VAEs in some internal metrics while falling short on others. To facilitate training we comprehensively explore the impact of common posterior collapse alleviation techniques in the literature. We release our code for reproducability.
We study a new problem of cross-lingual transfer learning for event coreference resolution (ECR) where models trained on data from a source language are adapted for evaluations in different target languages. We introduce the first baseline model for
this task based on XLM-RoBERTa, a state-of-the-art multilingual pre-trained language model. We also explore language adversarial neural networks (LANN) that present language discriminators to distinguish texts from the source and target languages to improve the language generalization for ECR. In addition, we introduce two novel mechanisms to further enhance the general representation learning of LANN, featuring: (i) multi-view alignment to penalize cross coreference-label alignment of examples in the source and target languages, and (ii) optimal transport to select close examples in the source and target languages to provide better training signals for the language discriminators. Finally, we perform extensive experiments for cross-lingual ECR from English to Spanish and Chinese to demonstrate the effectiveness of the proposed methods.
We investigate the representations learned by vision and language models in tasks that require relational reasoning. Focusing on the problem of assessing the relative size of objects in abstract visual contexts, we analyse both one-step and two-step
reasoning. For the latter, we construct a new dataset of three-image scenes and define a task that requires reasoning at the level of the individual images and across images in a scene. We probe the learned model representations using diagnostic classifiers. Our experiments show that pretrained multimodal transformer-based architectures can perform higher-level relational reasoning, and are able to learn representations for novel tasks and data that are very different from what was seen in pretraining.
This paper describes our contribution to the WASSA 2021 shared task on Empathy Prediction and Emotion Classification. The broad goal of this task was to model an empathy score, a distress score and the overall level of emotion of an essay written in
response to a newspaper article associated with harm to someone. We have used the ELECTRA model abundantly and also advanced deep learning approaches like multi-task learning. Additionally, we also leveraged standard machine learning techniques like ensembling. Our system achieves a Pearson Correlation Coefficient of 0.533 on sub-task I and a macro F1 score of 0.5528 on sub-task II. We ranked 1st in Emotion Classification sub-task and 3rd in Empathy Prediction sub-task.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
