Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userEnriching Transformers with Structured Tensor-Product Representations for Abstractive Summarization
تخصيب المحولات بتمثيلات منتجات العمال المهيكلة لتلخيص مبادرة
392
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تلخيص الجماعي، مهمة توليد ملخص موجز لمستندات المدخلات، يتطلب: (1) التفكير في المستند المصدر لتحديد القطع البارزة من المعلومات المنتشرة عبر المستند الطويل، و (2) تأليف نص متماسك بإعادة بناء هذه الحقائق البارزة في ملخص أقصر يعكس بإخلاص العلاقات المعقدة التي تربط هذه الحقائق. في هذه الورقة، نتكيف مع TP-Transformer (Schlag et al.، 2019)، وهي عبارة عن بنية تثري المحولات الأصلية (Vaswani et al.، 2017) مع تمثيل المنتج التركيبي الصريح للتوتر (TPR)، لمهمة التلخيص المبشور وبعد الميزة الرئيسية لطرازنا هي التحيز الهيكلي الذي نقدمه من خلال ترميز مجموعتين منفصلين لكل رمزي لتمثيل الهيكل النحوي (مع ناقلات الدور) والمحتوى الدلالي (مع ناقلات حشو) بشكل منفصل. ثم يربط النموذج ثم متجاهلة الدور والحشو إلى TPR كإخراج الطبقة. نقول أن التمثيلات الوسيطة المنظمة تمكن النموذج من السيطرة بشكل أفضل على المحتويات (الحقائق البارزة) والهياكل (بناء الجملة الذي يربط الحقائق) عند إنشاء الملخص. وإظهار تجريبيا، نظرا لأن لدينا TP-Transforment تفوقنا على المحول ومحول TP الأصلي بشكل كبير على العديد من مجموعات بيانات تلخيص الجماع على حد سواء التقييمات التلقائية والإنسانية. في العديد من المهام التحقيق النحوية واللالسة، نوضح المعلومات الهيكلية الناشئة في مجاهاجر الدور واكتساب الأداء من خلال خصوصية المعلومات من مواقف الدور وتحسين الترجمة الترجمة الترجمة النحوية في مخرجات طبقة TPR. (التعليمات البرمجية المتاحة في HTTPS: // github.com/jianggyctarheel/tpt-summ)
Abstractive summarization, the task of generating a concise summary of input documents, requires: (1) reasoning over the source document to determine the salient pieces of information scattered across the long document, and (2) composing a cohesive text by reconstructing these salient facts into a shorter summary that faithfully reflects the complex relations connecting these facts. In this paper, we adapt TP-Transformer (Schlag et al., 2019), an architecture that enriches the original Transformer (Vaswani et al., 2017) with the explicitly compositional Tensor Product Representation (TPR), for the task of abstractive summarization. The key feature of our model is a structural bias that we introduce by encoding two separate representations for each token to represent the syntactic structure (with role vectors) and semantic content (with filler vectors) separately. The model then binds the role and filler vectors into the TPR as the layer output. We argue that the structured intermediate representations enable the model to take better control of the contents (salient facts) and structures (the syntax that connects the facts) when generating the summary. Empirically, we show that our TP-Transformer outperforms the Transformer and the original TP-Transformer significantly on several abstractive summarization datasets based on both automatic and human evaluations. On several syntactic and semantic probing tasks, we demonstrate the emergent structural information in the role vectors and the performance gain by information specificity of the role vectors and improved syntactic interpretability in the TPR layer outputs.(Code and models are available at https://github.com/jiangycTarheel/TPT-Summ)
References used
https://aclanthology.org/
rate research
Read More
A crucial difference between single- and multi-document summarization is how salient content manifests itself in the document(s). While such content may appear at the beginning of a single document, essential information is frequently reiterated in a
set of documents related to a particular topic, resulting in an endorsement effect that increases information salience. In this paper, we model the cross-document endorsement effect and its utilization in multiple document summarization. Our method generates a synopsis from each document, which serves as an endorser to identify salient content from other documents. Strongly endorsed text segments are used to enrich a neural encoder-decoder model to consolidate them into an abstractive summary. The method has a great potential to learn from fewer examples to identify salient content, which alleviates the need for costly retraining when the set of documents is dynamically adjusted. Through extensive experiments on benchmark multi-document summarization datasets, we demonstrate the effectiveness of our proposed method over strong published baselines. Finally, we shed light on future research directions and discuss broader challenges of this task using a case study.
State-of-the-art abstractive summarization models generally rely on extensive labeled data, which lowers their generalization ability on domains where such data are not available. In this paper, we present a study of domain adaptation for the abstrac
tive summarization task across six diverse target domains in a low-resource setting. Specifically, we investigate the second phase of pre-training on large-scale generative models under three different settings: 1) source domain pre-training; 2) domain-adaptive pre-training; and 3) task-adaptive pre-training. Experiments show that the effectiveness of pre-training is correlated with the similarity between the pre-training data and the target domain task. Moreover, we find that continuing pre-training could lead to the pre-trained model's catastrophic forgetting, and a learning method with less forgetting can alleviate this issue. Furthermore, results illustrate that a huge gap still exists between the low-resource and high-resource settings, which highlights the need for more advanced domain adaptation methods for the abstractive summarization task.
Neural abstractive summarization systems have gained significant progress in recent years. However, abstractive summarization often produce inconsisitent statements or false facts. How to automatically generate highly abstract yet factually correct s
ummaries? In this paper, we proposed an efficient weak-supervised adversarial data augmentation approach to form the factual consistency dataset. Based on the artificial dataset, we train an evaluation model that can not only make accurate and robust factual consistency discrimination but is also capable of making interpretable factual errors tracing by backpropagated gradient distribution on token embeddings. Experiments and analysis conduct on public annotated summarization and factual consistency datasets demonstrate our approach effective and reasonable.
Abstract We introduce a simple but flexible mechanism to learn an intermediate plan to ground the generation of abstractive summaries. Specifically, we prepend (or prompt) target summaries with entity chains---ordered sequences of entities mentioned
in the summary. Transformer-based sequence-to-sequence models are then trained to generate the entity chain and then continue generating the summary conditioned on the entity chain and the input. We experimented with both pretraining and finetuning with this content planning objective. When evaluated on CNN/DailyMail, XSum, SAMSum, and BillSum, we demonstrate empirically that the grounded generation with the planning objective improves entity specificity and planning in summaries for all datasets, and achieves state-of-the-art performance on XSum and SAMSum in terms of rouge. Moreover, we demonstrate empirically that planning with entity chains provides a mechanism to control hallucinations in abstractive summaries. By prompting the decoder with a modified content plan that drops hallucinated entities, we outperform state-of-the-art approaches for faithfulness when evaluated automatically and by humans.
The paper describes a system for automatic summarization in English language of online news data that come from different non-English languages. The system is designed to be used in production environment for media monitoring. Automatic summarization
can be very helpful in this domain when applied as a helper tool for journalists so that they can review just the important information from the news channels. However, like every software solution, the automatic summarization needs performance monitoring and assured safe environment for the clients. In media monitoring environment the most problematic features to be addressed are: the copyright issues, the factual consistency, the style of the text and the ethical norms in journalism. Thus, the main contribution of our present work is that the above mentioned characteristics are successfully monitored in neural automatic summarization models and improved with the help of validation, fact-preserving and fact-checking procedures.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
