Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userThe GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
المؤشر جوهرة: توليد اللغة الطبيعية، تقييمها ومقاييسه
310
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
نحن نقدم جوهرة، معيار معيشة لتوليد اللغة الطبيعية (NLG)، تقييمه، ومقاييسه.تعتمد التقدم المحرز في NLG على نظام بيئي متطور باستمرار للمقاييس الآلية ومجموعات البيانات ومعايير التقييم البشري.نظرا لهذا الهدف المتحرك، لا تزال هناك نماذج جديدة غالبا ما لا تزال تقيمت في ولاية شركات الأنجلو المتداخلة مع مقاييس راسخة ولكنها معيبة ومقاييس.هذا الفصل يجعل من الصعب تحديد قيود النماذج والفرص الحالية للتقدم.تعالج GEM في معالجة هذه القيد هذه بيئة يمكن فيها تطبيق النماذج التي يمكن فيها تطبيقها بسهولة على مجموعة واسعة من المهام والتي يمكن اختبار استراتيجيات التقييم فيها.سيؤدي تحديثات منتظمة إلى المعيار إلى مساعدة أبحاث NLG على تصبح أكثر تعددا متعددة اللغات وتتطور التحدي إلى جانب النماذج.تعمل هذه الورقة كوصف للبيانات المهمة المشتركة 2021 في ورشة عمل GEM المرتبطة.
We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics. Measuring progress in NLG relies on a constantly evolving ecosystem of automated metrics, datasets, and human evaluation standards. Due to this moving target, new models often still evaluate on divergent anglo-centric corpora with well-established, but flawed, metrics. This disconnect makes it challenging to identify the limitations of current models and opportunities for progress. Addressing this limitation, GEM provides an environment in which models can easily be applied to a wide set of tasks and in which evaluation strategies can be tested. Regular updates to the benchmark will help NLG research become more multilingual and evolve the challenge alongside models. This paper serves as the description of the data for the 2021 shared task at the associated GEM Workshop.
References used
https://aclanthology.org/
rate research
Read More
The advent of Deep Learning and the availability of large scale datasets has accelerated research on Natural Language Generation with a focus on newer tasks and better models. With such rapid progress, it is vital to assess the extent of scientific p
rogress made and identify the areas/components that need improvement. To accomplish this in an automatic and reliable manner, the NLP community has actively pursued the development of automatic evaluation metrics. Especially in the last few years, there has been an increasing focus on evaluation metrics, with several criticisms of existing metrics and proposals for several new metrics. This tutorial presents the evolution of automatic evaluation metrics to their current state along with the emerging trends in this field by specifically addressing the following questions: (i) What makes NLG evaluation challenging? (ii) Why do we need automatic evaluation metrics? (iii) What are the existing automatic evaluation metrics and how can they be organised in a coherent taxonomy? (iv) What are the criticisms and shortcomings of existing metrics? (v) What are the possible future directions of research?
Knowledge-enriched text generation poses unique challenges in modeling and learning, driving active research in several core directions, ranging from integrated modeling of neural representations and symbolic information in the sequential/hierarchica
l/graphical structures, learning without direct supervisions due to the cost of structured annotation, efficient optimization and inference with massive and global constraints, to language grounding on multiple modalities, and generative reasoning with implicit commonsense knowledge and background knowledge. In this tutorial we will present a roadmap to line up the state-of-the-art methods to tackle these challenges on this cutting-edge problem. We will dive deep into various technical components: how to represent knowledge, how to feed knowledge into a generation model, how to evaluate generation results, and what are the remaining challenges?
We ask subjects whether they perceive as human-produced a bunch of texts, some of which are actually human-written, while others are automatically generated. We use this data to fine-tune a GPT-2 model to push it to generate more human-like texts, an
d observe that this fine-tuned model produces texts that are indeed perceived more human-like than the original model. Contextually, we show that our automatic evaluation strategy well correlates with human judgements. We also run a linguistic analysis to unveil the characteristics of human- vs machine-perceived language.
We propose an approach to automatically test for originality in generation tasks where no standard automatic measures exist. Our proposal addresses original uses of language, not necessarily original ideas. We provide an algorithm for our approach an
d a run-time analysis. The algorithm, which finds all of the original fragments in a ground-truth corpus and can reveal whether a generated fragment copies an original without attribution, has a run-time complexity of theta(nlogn) where n is the number of sentences in the ground truth.
Counterfactuals are a valuable means for understanding decisions made by ML systems. However, the counterfactuals generated by the methods currently available for natural language text are either unrealistic or introduce imperceptible changes. We pro
pose CounterfactualGAN: a method that combines a conditional GAN and the embeddings of a pretrained BERT encoder to model-agnostically generate realistic natural language text counterfactuals for explaining regression and classification tasks. Experimental results show that our method produces perceptibly distinguishable counterfactuals, while outperforming four baseline methods on fidelity and human judgments of naturalness, across multiple datasets and multiple predictive models.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
