Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userImproving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation
تحسين العلامة الصفرية والقليل من التلخيصات المخرجة مع ضبط الدقيقة المتوسطة وتكبير البيانات
269
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
يحقق النماذج المحددة ذات الأهداف الإشراف ذاتية الإشراف على النصوص الكبيرة على تحقيق أداء حديثة على مهام تلخيص النص الإنجليزية. ومع ذلك، فإن هذه النماذج عادة ما يتم ضبطها على مئات الآلاف من نقاط البيانات، ومتطلبات غير قابلة للتنفيذ عند تطبيق تلخيص لمجالات مخصصة جديدة. في هذا العمل، نقدم طريقة جديدة ومتعمول، تسمى Wikitransfer، للحصول على نماذج مطاطية دقيقة مسبقا للتلخيص بطريقة غير منشأة، خاصة بموجب البيانات. Wikitransfer Ground-Descrees نماذج مسبقا على الملخصات الزائفة، التي تم إنتاجها من بيانات Wikipedia العامة، والتي تحتوي على خصائص DataSet المستهدفة، مثل طول ومستوى التجريد من الملخصات المرغوبة. حقق نماذج Wikitransfer أداء موجات خارجي من بين الفن، وهو أداء التلخيص المبشط عن الصفر على مجموعة بيانات CNN-Dailymail وإظهار فعالية نهجنا على ثلاث مجموعات بيانات متنوعة إضافية. هذه النماذج هي أكثر قوة للبيانات الصاخبة وكذلك تحقيق أداء أقل أو قابلة للمقارنة قليلا باستخدام 10 و 100 أمثلة تدريبية عند مقارنة بنقل القليل من بيانات الملخصات الأخرى. لتعزيز الأداء، نؤدي إلى زيادة تكبير البيانات عبر الترجمة المستديرة الرحلة وكذلك إدخال مصطلح تنظيمي لتحسين تحويل القليل من الرصاص. لفهم دور جوانب البيانات في أداء النقل ونوعية ملخصات الإخراج الناتجة، فإننا ندرس تأثير مكونات مكونات بيانات الطاقة الدقيقة الخاصة بنا وتحليل الأداء القليل من الأداء باستخدام التقييم التلقائي والبشري.
Models pretrained with self-supervised objectives on large text corpora achieve state-of-the-art performance on English text summarization tasks. However, these models are typically fine-tuned on hundreds of thousands of data points, an infeasible requirement when applying summarization to new, niche domains. In this work, we introduce a novel and generalizable method, called WikiTransfer, for fine-tuning pretrained models for summarization in an unsupervised, dataset-specific manner. WikiTransfer fine-tunes pretrained models on pseudo-summaries, produced from generic Wikipedia data, which contain characteristics of the target dataset, such as the length and level of abstraction of the desired summaries. WikiTransfer models achieve state-of-the-art, zero-shot abstractive summarization performance on the CNN-DailyMail dataset and demonstrate the effectiveness of our approach on three additional diverse datasets. These models are more robust to noisy data and also achieve better or comparable few-shot performance using 10 and 100 training examples when compared to few-shot transfer from other summarization datasets. To further boost performance, we employ data augmentation via round-trip translation as well as introduce a regularization term for improved few-shot transfer. To understand the role of dataset aspects in transfer performance and the quality of the resulting output summaries, we further study the effect of the components of our unsupervised fine-tuning data and analyze few-shot performance using both automatic and human evaluation.
References used
https://aclanthology.org/
rate research
Read More
Supplementary Training on Intermediate Labeled-data Tasks (STILT) is a widely applied technique, which first fine-tunes the pretrained language models on an intermediate task before on the target task of interest. While STILT is able to further impro
ve the performance of pretrained language models, it is still unclear why and when it works. Previous research shows that those intermediate tasks involving complex inference, such as commonsense reasoning, work especially well for RoBERTa-large. In this paper, we discover that the improvement from an intermediate task could be orthogonal to it containing reasoning or other complex skills --- a simple real-fake discrimination task synthesized by GPT2 can benefit diverse target tasks. We conduct extensive experiments to study the impact of different factors on STILT. These findings suggest rethinking the role of intermediate fine-tuning in the STILT pipeline.
Neural topic models can augment or replace bag-of-words inputs with the learned representations of deep pre-trained transformer-based word prediction models. One added benefit when using representations from multilingual models is that they facilitat
e zero-shot polylingual topic modeling. However, while it has been widely observed that pre-trained embeddings should be fine-tuned to a given task, it is not immediately clear what supervision should look like for an unsupervised task such as topic modeling. Thus, we propose several methods for fine-tuning encoders to improve both monolingual and zero-shot polylingual neural topic modeling. We consider fine-tuning on auxiliary tasks, constructing a new topic classification task, integrating the topic classification objective directly into topic model training, and continued pre-training. We find that fine-tuning encoder representations on topic classification and integrating the topic classification task directly into topic modeling improves topic quality, and that fine-tuning encoder representations on any task is the most important factor for facilitating cross-lingual transfer.
Data-to-text (D2T) generation in the biomedical domain is a promising - yet mostly unexplored - field of research. Here, we apply neural models for D2T generation to a real-world dataset consisting of package leaflets of European medicines. We show t
hat fine-tuned transformers are able to generate realistic, multi-sentence text from data in the biomedical domain, yet have important limitations. We also release a new dataset (BioLeaflets) for benchmarking D2T generation models in the biomedical domain.
While multilingual pretrained language models (LMs) fine-tuned on a single language have shown substantial cross-lingual task transfer capabilities, there is still a wide performance gap in semantic parsing tasks when target language supervision is a
vailable. In this paper, we propose a novel Translate-and-Fill (TaF) method to produce silver training data for a multilingual semantic parser. This method simplifies the popular Translate-Align-Project (TAP) pipeline and consists of a sequence-to-sequence filler model that constructs a full parse conditioned on an utterance and a view of the same parse. Our filler is trained on English data only but can accurately complete instances in other languages (i.e., translations of the English training utterances), in a zero-shot fashion. Experimental results on three multilingual semantic parsing datasets show that data augmentation with TaF reaches accuracies competitive with similar systems which rely on traditional alignment techniques.
This paper explores three simple data manipulation techniques (synthesis, augmentation, curriculum) for improving abstractive summarization models without the need for any additional data. We introduce a method of data synthesis with paraphrasing, a
data augmentation technique with sample mixing, and curriculum learning with two new difficulty metrics based on specificity and abstractiveness. We conduct experiments to show that these three techniques can help improve abstractive summarization across two summarization models and two different small datasets. Furthermore, we show that these techniques can improve performance when applied in isolation and when combined.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
