Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userEstimating Subjective Crowd-Evaluations as an Additional Objective to Improve Natural Language Generation
تقدير تقييمات الحشود الذاتية كهدف إضافي لتحسين توليد اللغة الطبيعية
245
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تعد التصنيفات البشرية واحدة من أكثر الطرق سائدة لتقييم أداء خوارزميات NLP (معالجة اللغة الطبيعية). وبالمثل، من الشائع قياس جودة الجمل الناتجة عن نموذج توليد اللغة الطبيعي باستخدام الدراسات البشرية. في هذه الورقة، نقول لاستكشاف استخدام التقييمات الذاتية ضمن عملية نماذج توليد اللغة التدريب في إعداد تعليمي متعدد المهام. كدراسة حالة، نستخدم كوربا حوارا مؤلفا من الحشود لضبط ستة نماذج توليد لغة مختلفة. يتضمن اثنان من هذه النماذج تعلم المهام المتعددة واستخدام التصنيفات الذاتية للخطوط كجزء من هدف التعلم الصريح. تكشف التقييم البشري لخطوط الحوار التي تم إنشاؤها أن الكلام الناتجة عن النماذج متعددة المهام تم تصنيفها بشكل ذاتي باعتبارها الأكثر نموذجية، وتحريك المحادثة للأمام، وأقل هجومية. استنادا إلى النتائج الأولى الواعدة، نقوم بمناقشة اتجاهات البحث في المستقبل لدمج التقييمات الإنسانية الذاتية في التدريب النموذجي اللغوي وبالتالي الحفاظ على المستخدم البشري في الحلقة أثناء عملية التطوير.
Human ratings are one of the most prevalent methods to evaluate the performance of NLP (natural language processing) algorithms. Similarly, it is common to measure the quality of sentences generated by a natural language generation model using human raters. In this paper we argue for exploring the use of subjective evaluations within the process of training language generation models in a multi-task learning setting. As a case study, we use a crowd-authored dialogue corpus to fine-tune six different language generation models. Two of these models incorporate multi-task learning and use subjective ratings of lines as part of an explicit learning goal. A human evaluation of the generated dialogue lines reveals that utterances generated by the multi-tasking models were subjectively rated as the most typical, most moving the conversation forward, and least offensive. Based on these promising first results, we discuss future research directions for incorporating subjective human evaluations into language model training and to hence keep the human user in the loop during the development process.
References used
https://aclanthology.org/
rate research
Read More
Knowledge-enriched text generation poses unique challenges in modeling and learning, driving active research in several core directions, ranging from integrated modeling of neural representations and symbolic information in the sequential/hierarchica
l/graphical structures, learning without direct supervisions due to the cost of structured annotation, efficient optimization and inference with massive and global constraints, to language grounding on multiple modalities, and generative reasoning with implicit commonsense knowledge and background knowledge. In this tutorial we will present a roadmap to line up the state-of-the-art methods to tackle these challenges on this cutting-edge problem. We will dive deep into various technical components: how to represent knowledge, how to feed knowledge into a generation model, how to evaluate generation results, and what are the remaining challenges?
We ask subjects whether they perceive as human-produced a bunch of texts, some of which are actually human-written, while others are automatically generated. We use this data to fine-tune a GPT-2 model to push it to generate more human-like texts, an
d observe that this fine-tuned model produces texts that are indeed perceived more human-like than the original model. Contextually, we show that our automatic evaluation strategy well correlates with human judgements. We also run a linguistic analysis to unveil the characteristics of human- vs machine-perceived language.
We propose an approach to automatically test for originality in generation tasks where no standard automatic measures exist. Our proposal addresses original uses of language, not necessarily original ideas. We provide an algorithm for our approach an
d a run-time analysis. The algorithm, which finds all of the original fragments in a ground-truth corpus and can reveal whether a generated fragment copies an original without attribution, has a run-time complexity of theta(nlogn) where n is the number of sentences in the ground truth.
In this paper, we study the utilization of pre-trained language models to enable few-shotNatural Language Generation (NLG) in task-oriented dialog systems. We introduce a system consisting of iterative self-training and an extensible mini-template fr
amework that textualizes the structured input data into semi-natural text to fully take advantage of pre-trained language models. We compare var-ious representations of NLG models' input and output and show that transforming the input and output to be similar to what the language model has seen before during pre-training improves the model's few-shot performance substantially. We show that neural mod-els can be trained with as few as 300 annotated examples while providing high fidelity, considerably lowering the resource requirements for standing up a new domain or language.This level of data efficiency removes the need for crowd-sourced data collection resulting in higher quality data annotated by expert linguists. In addition, model maintenance and debugging processes will improve in this few-shot setting. Finally, we explore distillation and using a caching system to satisfy latency requirements of real-world systems.
We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this m
oving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for the 2021 shared task at the associated GEM Workshop.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
