Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userResults of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain
نتائج مقاييس WMT21 المهمة المشتركة: تقييم المقاييس مع التقييمات البشرية القائمة على الخبراء على نطاق TED و TEWS
338
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تقدم هذه الورقة نتائج المهمة المشتركة للمقاييس WMT21. طلب من المشاركين تسجيل مخرجات أنظمة الترجمة المتنافسة في مهمة الترجمة الأخبار WMT21 مع مقاييس أوتوماتيكية على مجطتين مختلفتين: أخبار ومحادثات تيد. تم تقييم جميع المقاييس على مدى ارتباطها على مستوى النظام والقطاع مع التصنيفات البشرية. على عكس إصدارات السنوات السابقة، فقد استحوذنا هذا العام على تصنيفاتنا الإنسانية الخاصة بنا على أساس التقييم البشري القائم على الخبراء عبر مقاييس الجودة متعددة الأبعاد (MQM). يحتوي هذا الإعداد على العديد من المزايا: (1) قد أظهر التقييم القائم على الخبراء أنه أكثر موثوقية، (2) تمكنا من تقييم جميع المقاييس على مجاليين مختلفين باستخدام ترجمات نفس أنظمة MT، (III) أضفنا 5 إضافية الترجمات القادمة من نفس النظام أثناء تطوير النظام. بالإضافة إلى ذلك، صممنا ثلاث مجموعات تحدي تقيم متانة جميع المقاييس التلقائية. نقدم تحليلا مكثفا حول مدى أداء المقاييس على أزواج ثلاث لغات: الإنجليزية إلى الألمانية والإنجليزية إلى الروسية والصينية إلى الإنجليزية. نوضح تأثير الترجمات المرجعية المختلفة على المقاييس المستندة إلى المرجع ومقارنة شرح MQM القائم على الخبراء مع درجات DA المكتسبة بواسطة WMT.
This paper presents the results of the WMT21 Metrics Shared Task. Participants were asked to score the outputs of the translation systems competing in the WMT21 News Translation Task with automatic metrics on two different domains: news and TED talks. All metrics were evaluated on how well they correlate at the system- and segment-level with human ratings. Contrary to previous years' editions, this year we acquired our own human ratings based on expert-based human evaluation via Multidimensional Quality Metrics (MQM). This setup had several advantages: (i) expert-based evaluation has been shown to be more reliable, (ii) we were able to evaluate all metrics on two different domains using translations of the same MT systems, (iii) we added 5 additional translations coming from the same system during system development. In addition, we designed three challenge sets that evaluate the robustness of all automatic metrics. We present an extensive analysis on how well metrics perform on three language pairs: English to German, English to Russian and Chinese to English. We further show the impact of different reference translations on reference-based metrics and compare our expert-based MQM annotation with the DA scores acquired by WMT.
References used
https://aclanthology.org/
rate research
Read More
In this paper, we describe our submission to the WMT 2021 Metrics Shared Task. We use the automatically-generated questions and answers to evaluate the quality of Machine Translation (MT) systems. Our submission builds upon the recently proposed MTEQ
A framework. Experiments on WMT20 evaluation datasets show that at the system-level the MTEQA metric achieves performance comparable with other state-of-the-art solutions, while considering only a certain amount of information from the whole translation.
The NLP field has recently seen a substantial increase in work related to reproducibility of results, and more generally in recognition of the importance of having shared definitions and practices relating to evaluation. Much of the work on reproduci
bility has so far focused on metric scores, with reproducibility of human evaluation results receiving far less attention. As part of a research programme designed to develop theory and practice of reproducibility assessment in NLP, we organised the first shared task on reproducibility of human evaluations, ReproGen 2021. This paper describes the shared task in detail, summarises results from each of the reproduction studies submitted, and provides further comparative analysis of the results. Out of nine initial team registrations, we received submissions from four teams. Meta-analysis of the four reproduction studies revealed varying degrees of reproducibility, and allowed very tentative first conclusions about what types of evaluation tend to have better reproducibility.
The Shared Task on Evaluating Accuracy focused on techniques (both manual and automatic) for evaluating the factual accuracy of texts produced by neural NLG systems, in a sports-reporting domain. Four teams submitted evaluation techniques for this ta
sk, using very different approaches and techniques. The best-performing submissions did encouragingly well at this difficult task. However, all automatic submissions struggled to detect factual errors which are semantically or pragmatically complex (for example, based on incorrect computation or inference).
This paper describes Papago submission to the WMT 2021 Quality Estimation Task 1: Sentence-level Direct Assessment. Our multilingual Quality Estimation system explores the combination of Pretrained Language Models and Multi-task Learning architecture
s. We propose an iterative training pipeline based on pretraining with large amounts of in-domain synthetic data and finetuning with gold (labeled) data. We then compress our system via knowledge distillation in order to reduce parameters yet maintain strong performance. Our submitted multilingual systems perform competitively in multilingual and all 11 individual language pair settings including zero-shot.
This paper describes Charles University sub-mission for Terminology translation Shared Task at WMT21. The objective of this task is to design a system which translates certain terms based on a provided terminology database, while preserving high over
all translation quality. We competed in English-French language pair. Our approach is based on providing the desired translations alongside the input sentence and training the model to use these provided terms. We lemmatize the terms both during the training and inference, to allow the model to learn how to produce correct surface forms of the words, when they differ from the forms provided in the terminology database. Our submission ranked second in Exact Match metric which evaluates the ability of the model to produce desired terms in the translation.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
