Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userGeSERA: General-domain Summary Evaluation by Relevance Analysis
Gesera: تقييم ملخص المجال العام عن طريق تحليل الصلة
465
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
نقوم بإصدار Gesera، وهي نسخة محسنة مفتوحة المصدر من SERA لتقييم الملخصات الاستخراجية والتغييرات التلقائية من المجال العام. تعتمد SERA على محرك بحث يقارن الملخصات المرشحة والمرجعية (تسمى الاستعلامات) مقابل قاعدة مستندات استرجاع المعلومات (تسمى المؤشر). تم تصميم Sera في الأصل للنطاق الطبي الطبيعي فقط، حيث أظهرت ارتباطا أفضل مع الأساليب اليدوية من طريقة Rouge المستندة إلى المعجمات المستخدمة على نطاق واسع. في هذه الورقة، نأخذ سيرا من المجال الطبي الطبيعي إلى عام واحد من خلال تكييف أسلوبها القائم على المحتوى لتقييم الملخصات بنجاح من المجال العام. أولا، نحسن استراتيجية إعادة صياغة الاستعلام مع تحليل علامات نقاط البيع لعوريا المجال العام. ثانيا، نستبدل المؤشر الطبي الحيوي المستخدم في سيرا بمجموعات مقالة مقالة من مادتين ويكيبيديا. نحن نقوم بإجراء تجارب مع مجموعات بيانات TAC2008 و TAC2009 و CNNDM. تظهر النتائج أنه في معظم الحالات، تحقق Gesera ارتباطا أعلى مع أساليب التقييم اليدوي من SERA، بينما يقلل من فجوا لها مع Rouge for General-Domain Definuation. حتى gesera حتى يتجاوز الحمر في حالتين من TAC2009. أخيرا، نقوم بإجراء تجارب مكثفة وتوفير دراسة شاملة لتأثير الحنجرة البشرية وحجم المؤشر على التقييم الموجز مع SERA و GESERA.
We present GeSERA, an open-source improved version of SERA for evaluating automatic extractive and abstractive summaries from the general domain. SERA is based on a search engine that compares candidate and reference summaries (called queries) against an information retrieval document base (called index). SERA was originally designed for the biomedical domain only, where it showed a better correlation with manual methods than the widely used lexical-based ROUGE method. In this paper, we take out SERA from the biomedical domain to the general one by adapting its content-based method to successfully evaluate summaries from the general domain. First, we improve the query reformulation strategy with POS Tags analysis of general-domain corpora. Second, we replace the biomedical index used in SERA with two article collections from AQUAINT-2 and Wikipedia. We conduct experiments with TAC2008, TAC2009, and CNNDM datasets. Results show that, in most cases, GeSERA achieves higher correlations with manual evaluation methods than SERA, while it reduces its gap with ROUGE for general-domain summary evaluation. GeSERA even surpasses ROUGE in two cases of TAC2009. Finally, we conduct extensive experiments and provide a comprehensive study of the impact of human annotators and the index size on summary evaluation with SERA and GeSERA.
References used
https://aclanthology.org/
rate research
Read More
Hate speech detection is an actively growing field of research with a variety of recently proposed approaches that allowed to push the state-of-the-art results. One of the challenges of such automated approaches -- namely recent deep learning models
-- is a risk of false positives (i.e., false accusations), which may lead to over-blocking or removal of harmless social media content in applications with little moderator intervention. We evaluate deep learning models both under in-domain and cross-domain hate speech detection conditions, and introduce an SVM approach that allows to significantly improve the state-of-the-art results when combined with the deep learning models through a simple majority-voting ensemble. The improvement is mainly due to a reduction of the false positive rate.
We propose a new reference-free summary quality evaluation measure, with emphasis on the faithfulness. The measure is based on finding and counting all probable potential inconsistencies of the summary with respect to the source document. The propose
d ESTIME, Estimator of Summary-to-Text Inconsistency by Mismatched Embeddings, correlates with expert scores in summary-level SummEval dataset stronger than other common evaluation measures not only in Consistency but also in Fluency. We also introduce a method of generating subtle factual errors in human summaries. We show that ESTIME is more sensitive to subtle errors than other common evaluation measures.
We introduce a new dataset for Question Rewriting in Conversational Context (QReCC), which contains 14K conversations with 80K question-answer pairs. The task in QReCC is to find answers to conversational questions within a collection of 10M web page
s (split into 54M passages). Answers to questions in the same conversation may be distributed across several web pages. QReCC provides annotations that allow us to train and evaluate individual subtasks of question rewriting, passage retrieval and reading comprehension required for the end-to-end conversational question answering (QA) task. We report the effectiveness of a strong baseline approach that combines the state-of-the-art model for question rewriting, and competitive models for open-domain QA. Our results set the first baseline for the QReCC dataset with F1 of 19.10, compared to the human upper bound of 75.45, indicating the difficulty of the setup and a large room for improvement.
We introduce SPARTA, a novel neural retrieval method that shows great promise in performance, generalization, and interpretability for open-domain question answering. Unlike many neural ranking methods that use dense vector nearest neighbor search, S
PARTA learns a sparse representation that can be efficiently implemented as an Inverted Index. The resulting representation enables scalable neural retrieval that does not require expensive approximate vector search and leads to better performance than its dense counterpart. We validated our approaches on 4 open-domain question answering (OpenQA) tasks and 11 retrieval question answering (ReQA) tasks. SPARTA achieves new state-of-the-art results across a variety of open-domain question answering tasks in both English and Chinese datasets, including open SQuAD, CMRC and etc. Analysis also confirms that the proposed method creates human interpretable representation and allows flexible control over the trade-off between performance and efficiency.
Many NLG tasks such as summarization, dialogue response, or open domain question answering, focus primarily on a source text in order to generate a target response. This standard approach falls short, however, when a user's intent or context of work
is not easily recoverable based solely on that source text-- a scenario that we argue is more of the rule than the exception. In this work, we argue that NLG systems in general should place a much higher level of emphasis on making use of additional context, and suggest that relevance (as used in Information Retrieval) be thought of as a crucial tool for designing user-oriented text-generating tasks. We further discuss possible harms and hazards around such personalization, and argue that value-sensitive design represents a crucial path forward through these challenges.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
