Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userThe Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
مخاطر استخدام الترك الميكانيكي لتقييم جيل النص مفتوح العضوية
306
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
ركزت أبحاث جيل النص الحديثة بشكل متزايد على المجالات المفتوحة مثل القصة وتوليد الشعر. نظرا لأن النماذج التي تم بناؤها لهذه المهام يصعب تقييمها تلقائيا، يبرر معظم الباحثين في الفضاء خيارات النمذجة عن طريق جمع الأحكام البشرية الجماعية لجودة النص (على سبيل المثال، عشرات Likert من الاتساق أو النحاس) من Amazon Mechanical Turk (AMT). في هذه الورقة، نقوم أولا بإجراء دراسة استقصائية قدرها 45 ورقة من جيل النص مفتوح العضوية وتجد أن الغالبية العظمى منهم يفشلون في الإبلاغ عن تفاصيل حاسمة حول مهام AMT الخاصة بهم، مما أعاق الاستيلاء. بعد ذلك، قم بتشغيل سلسلة من تجارب تقييم القصة مع كل من عمال AMT ومعلمي اللغة الإنجليزية واكتشف أنه حتى مع مرشحات التأهيل الصارمة، يفشل عمال AMT (على عكس المعلمين) في التمييز بين النص النموذجي والمراجع التي تم إنشاؤها الإنسان. نظهر أن أحكام عامل AMT تتحسن عند إظهار الناتج الناتج عن النموذج إلى جانب المراجع التي يتم إنشاؤها من قبل الإنسان، مما يتيح العمال معا معايرة تقييماتهم بشكل أفضل. أخيرا، توفر المقابلات مع معلمي اللغة الإنجليزية رؤى أعمق في تحديات عملية التقييم، خاصة عند تصنيف النص الذي تم إنشاؤه النموذج.
Recent text generation research has increasingly focused on open-ended domains such as story and poetry generation. Because models built for such tasks are difficult to evaluate automatically, most researchers in the space justify their modeling choices by collecting crowdsourced human judgments of text quality (e.g., Likert scores of coherence or grammaticality) from Amazon Mechanical Turk (AMT). In this paper, we first conduct a survey of 45 open-ended text generation papers and find that the vast majority of them fail to report crucial details about their AMT tasks, hindering reproducibility. We then run a series of story evaluation experiments with both AMT workers and English teachers and discover that even with strict qualification filters, AMT workers (unlike teachers) fail to distinguish between model-generated text and human-generated references. We show that AMT worker judgments improve when they are shown model-generated output alongside human-generated references, which enables the workers to better calibrate their ratings. Finally, interviews with the English teachers provide deeper insights into the challenges of the evaluation process, particularly when rating model-generated text.
References used
https://aclanthology.org/
rate research
Read More
Current commonsense reasoning research focuses on developing models that use commonsense knowledge to answer multiple-choice questions. However, systems designed to answer multiple-choice questions may not be useful in applications that do not provid
e a small list of candidate answers to choose from. As a step towards making commonsense reasoning research more realistic, we propose to study open-ended commonsense reasoning (OpenCSR) --- the task of answering a commonsense question without any pre-defined choices --- using as a resource only a corpus of commonsense facts written in natural language. OpenCSR is challenging due to a large decision space, and because many questions require implicit multi-hop reasoning. As an approach to OpenCSR, we propose DrFact, an efficient Differentiable model for multi-hop Reasoning over knowledge Facts. To evaluate OpenCSR methods, we adapt several popular commonsense reasoning benchmarks, and collect multiple new answers for each test question via crowd-sourcing. Experiments show that DrFact outperforms strong baseline methods by a large margin.
We present DART, an open domain structured DAta Record to Text generation dataset with over 82k instances (DARTs). Data-to-text annotations can be a costly process, especially when dealing with tables which are the major source of structured data and
contain nontrivial structures. To this end, we propose a procedure of extracting semantic triples from tables that encodes their structures by exploiting the semantic dependencies among table headers and the table title. Our dataset construction framework effectively merged heterogeneous sources from open domain semantic parsing and spoken dialogue systems by utilizing techniques including tree ontology annotation, question-answer pair to declarative sentence conversion, and predicate unification, all with minimum post-editing. We present systematic evaluation on DART as well as new state-of-the-art results on WebNLG 2017 to show that DART (1) poses new challenges to existing data-to-text datasets and (2) facilitates out-of-domain generalization. Our data and code can be found at https://github.com/Yale-LILY/dart.
Data-to-text generation systems are trained on large datasets, such as WebNLG, Ro-toWire, E2E or DART. Beyond traditional token-overlap evaluation metrics (BLEU or METEOR), a key concern faced by recent generators is to control the factuality of the
generated text with respect to the input data specification. We report on our experience when developing an automatic factuality evaluation system for data-to-text generation that we are testing on WebNLG and E2E data. We aim to prepare gold data annotated manually to identify cases where the text communicates more information than is warranted based on the in-put data (extra) or fails to communicate data that is part of the input (missing). While analyzing reference (data, text) samples, we encountered a range of systematic uncertainties that are related to cases on implicit phenomena in text, and the nature of non-linguistic knowledge we expect to be involved when assessing factuality. We derive from our experience a set of evaluation guidelines to reach high inter-annotator agreement on such cases.
In this paper, we introduce a new embedding-based metric relying on trainable ranking models to evaluate the semantic accuracy of neural data-to-text generators. This metric is especially well suited to semantically and factually assess the performan
ce of a text generator when tables can be associated with multiple references and table values contain textual utterances. We first present how one can implement and further specialize the metric by training the underlying ranking models on a legal Data-to-Text dataset. We show how it may provide a more robust evaluation than other evaluation schemes in challenging settings using a dataset comprising paraphrases between the table values and their respective references. Finally, we evaluate its generalization capabilities on a well-known dataset, WebNLG, by comparing it with human evaluation and a recently introduced metric based on natural language inference. We then illustrate how it naturally characterizes, both quantitatively and qualitatively, omissions and hallucinations.
The analytical description of charts is an exciting and important research area with many applications in academia and industry. Yet, this challenging task has received limited attention from the computational linguistics research community. This pap
er proposes AutoChart, a large dataset for the analytical description of charts, which aims to encourage more research into this important area. Specifically, we offer a novel framework that generates the charts and their analytical description automatically. We conducted extensive human and machine evaluation on the generated charts and descriptions and demonstrate that the generated texts are informative, coherent, and relevant to the corresponding charts.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
