Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userFinding a Balanced Degree of Automation for Summary Evaluation
العثور على درجة متوازنة من الأتمتة لتقييم الموجز
136
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
التقييم البشري لمهام التلخيص موثوقة ولكن يجلب قضايا التكاثر والتكاليف العالية. المقاييس التلقائية رخيصة وغير قابلة للتكرار ولكن في بعض الأحيان ترتبط بشكل سيء بحكمات بشرية. في هذا العمل، نقترح Nemiautomatic مرنة لمقاييس التقييم الموجز التلقائي، بعد طريقة التقييم البشري الهرم. يحتفظ Lite2Pyramid شبه التلقائي بوحدات المحتوى الموجزة ذات العلامة البشرية القابلة لإعادة الاستخدام (SCU) للإشارة (SCU)، لكنها تحل محل العمل اليدوي للحكم على وجود قاضم في ملخصات النظام مع نموذج استنتاج اللغة الطبيعية (NLI). تستبدل Lite3pyramid التلقائي بالكامل مزيد من البدائل SCUS مع الوحدات الثلاثية الدلالية المستخرجة تلقائيا (STUS) عبر نموذج العلامات الدلالية (SRL). أخيرا، نقترح مقاييس، Lite2.xpyramid، حيث نستخدم نموذجا بسيطا للتنبؤ بمدى محاكاة STUS محاكاة SCUS والاحتفاظ ب SCUs الأكثر صعوبة في محاكاة، والتي توفر عملية انتقال سلسة وتوازن بين الأتمتة والتقييم اليدوي وبعد مقارنة 15 مقاييس موجودة، نقوم بتقييم الارتباطات المترية البشرية على 3 مجموعات بيانات تقييم التلوث الحالية و Pyrxsum التي تم جمعها حديثا (مع أمثلة / أنظمة / أنظمة 100/10 XSUM). يظهر أن Lite2Pyramid لديها باستمرار أفضل الارتباطات على مستوى الملخص؛ يعمل Lite3pyramid بشكل أفضل من أو قابلة للمقارنة مع مقاييس أوتوماتيكية أخرى؛ يتداول Lite2.XPyramID قبالة قطرات الارتباط الصغيرة لخفض الجهد اليدوي الأكبر، والتي يمكن أن تقلل من تكاليف جمع البيانات المستقبلية.
Human evaluation for summarization tasks is reliable but brings in issues of reproducibility and high costs. Automatic metrics are cheap and reproducible but sometimes poorly correlated with human judgment. In this work, we propose flexible semiautomatic to automatic summary evaluation metrics, following the Pyramid human evaluation method. Semi-automatic Lite2Pyramid retains the reusable human-labeled Summary Content Units (SCUs) for reference(s) but replaces the manual work of judging SCUs' presence in system summaries with a natural language inference (NLI) model. Fully automatic Lite3Pyramid further substitutes SCUs with automatically extracted Semantic Triplet Units (STUs) via a semantic role labeling (SRL) model. Finally, we propose in-between metrics, Lite2.xPyramid, where we use a simple regressor to predict how well the STUs can simulate SCUs and retain SCUs that are more difficult to simulate, which provides a smooth transition and balance between automation and manual evaluation. Comparing to 15 existing metrics, we evaluate human-metric correlations on 3 existing meta-evaluation datasets and our newly collected PyrXSum (with 100/10 XSum examples/systems). It shows that Lite2Pyramid consistently has the best summary-level correlations; Lite3Pyramid works better than or comparable to other automatic metrics; Lite2.xPyramid trades off small correlation drops for larger manual effort reduction, which can reduce costs for future data collection.
References used
https://aclanthology.org/
rate research
Read More
Many applications require generation of summaries tailored to the user's information needs, i.e., their intent. Methods that express intent via explicit user queries fall short when query interpretation is subjective. Several datasets exist for summa
rization with objective intents where, for each document and intent (e.g., weather''), a single summary suffices for all users. No datasets exist, however, for subjective intents (e.g., interesting places'') where different users will provide different summaries. We present SUBSUME, the first dataset for evaluation of SUBjective SUMmary Extraction systems. SUBSUME contains 2,200 (document, intent, summary) triplets over 48 Wikipedia pages, with ten intents of varying subjectivity, provided by 103 individuals over Mechanical Turk. We demonstrate statistically that the intents in SUBSUME vary systematically in subjectivity. To indicate SUBSUME's usefulness, we explore a collection of baseline algorithms for subjective extractive summarization and show that (i) as expected, example-based approaches better capture subjective intents than query-based ones, and (ii) there is ample scope for improving upon the baseline algorithms, thereby motivating further research on this challenging problem.
Sentence embeddings encode information relating to the usage of idioms in a sentence. This paper reports a set of experiments that combine a probing methodology with input masking to analyse where in a sentence this idiomatic information is taken fro
m, and what form it takes. Our results indicate that BERT's idiomatic key is primarily found within an idiomatic expression, but also draws on information from the surrounding context. Also, BERT can distinguish between the disruption in a sentence caused by words missing and the incongruity caused by idiomatic usage.
Scholarly documents have a great degree of variation, both in terms of content (semantics) and structure (pragmatics). Prior work in scholarly document understanding emphasizes semantics through document summarization and corpus topic modeling but te
nds to omit pragmatics such as document organization and flow. Using a corpus of scholarly documents across 19 disciplines and state-of-the-art language modeling techniques, we learn a fixed set of domain-agnostic descriptors for document sections and retrofit'' the corpus to these descriptors (also referred to as normalization''). Then, we analyze the position and ordering of these descriptors across documents to understand the relationship between discipline and structure. We report within-discipline structural archetypes, variability, and between-discipline comparisons, supporting the hypothesis that scholarly communities, despite their size, diversity, and breadth, share similar avenues for expressing their work. Our findings lay the foundation for future work in assessing research quality, domain style transfer, and further pragmatic analysis.
Abstract A desirable property of a reference-based evaluation metric that measures the content quality of a summary is that it should estimate how much information that summary has in common with a reference. Traditional text overlap based metrics su
ch as ROUGE fail to achieve this because they are limited to matching tokens, either lexically or via embeddings. In this work, we propose a metric to evaluate the content quality of a summary using question-answering (QA). QA-based methods directly measure a summary's information overlap with a reference, making them fundamentally different than text overlap metrics. We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval. QAEval outperforms current state-of-the-art metrics on most evaluations using benchmark datasets, while being competitive on others due to limitations of state-of-the-art models. Through a careful analysis of each component of QAEval, we identify its performance bottlenecks and estimate that its potential upper-bound performance surpasses all other automatic metrics, approaching that of the gold-standard Pyramid Method.1
The dominant approach in probing neural networks for linguistic properties is to train a new shallow multi-layer perceptron (MLP) on top of the model's internal representations. This approach can detect properties encoded in the model, but at the cos
t of adding new parameters that may learn the task directly. We instead propose a subtractive pruning-based probe, where we find an existing subnetwork that performs the linguistic task of interest. Compared to an MLP, the subnetwork probe achieves both higher accuracy on pre-trained models and lower accuracy on random models, so it is both better at finding properties of interest and worse at learning on its own. Next, by varying the complexity of each probe, we show that subnetwork probing Pareto-dominates MLP probing in that it achieves higher accuracy given any budget of probe complexity. Finally, we analyze the resulting subnetworks across various tasks to locate where each task is encoded, and we find that lower-level tasks are captured in lower layers, reproducing similar findings in past work.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
