Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userExperts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation
الخبراء والأخطاء والسياق: دراسة واسعة النطاق للتقييم البشري للترجمة الآلية
574
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
التقييم البشري التجريدي لأنظمة الترجمة عالية الجودة الحديثة هي مشكلة صعبة، وهناك أدلة متزايدة على أن إجراءات التقييم غير الكافية يمكن أن تؤدي إلى استنتاجات خاطئة. بينما كان هناك بحث كبير في التقييم البشري، لا يزال الحقل يفتقر إلى إجراء قياسي شائع. كخطوة نحو هذا الهدف، نقترح منهجية تقييم في تحليل خطأ صريح، استنادا إلى إطار مقاييس الجودة متعددة الأبعاد (MQM). نحن نفذت أكبر دراسة بحثية MQM حتى الآن، وتسجيل مخرجات الأنظمة العليا من المهمة المشتركة WMT 2020 في أزواج لغتين باستخدام التعليقات التوضيحية المقدمة من المترجمين المحترفين مع الوصول إلى سياق المستند الكامل. نقوم بتحليل البيانات الناتجة على نطاق واسع، والعثور على نتائج أخرى بمثابة ترتيب مختلف تماما للأنظمة المقدرة من تلك المنشأة من قبل عمال الحشد WMT، تعرض تفضيل واضح لإخراج الإنسان على الجهاز. من المستغرب، نجد أيضا أن المقاييس التلقائية القائمة على المدينات المدربة مسبقا يمكن أن تفوق عمال الحشد البشري. نحن نجعل كوربوس متاحة علنا لمزيد من البحث.
Abstract Human evaluation of modern high-quality machine translation systems is a difficult problem, and there is increasing evidence that inadequate evaluation procedures can lead to erroneous conclusions. While there has been considerable research on human evaluation, the field still lacks a commonly accepted standard procedure. As a step toward this goal, we propose an evaluation methodology grounded in explicit error analysis, based on the Multidimensional Quality Metrics (MQM) framework. We carry out the largest MQM research study to date, scoring the outputs of top systems from the WMT 2020 shared task in two language pairs using annotations provided by professional translators with access to full document context. We analyze the resulting data extensively, finding among other results a substantially different ranking of evaluated systems from the one established by the WMT crowd workers, exhibiting a clear preference for human over machine output. Surprisingly, we also find that automatic metrics based on pre-trained embeddings can outperform human crowd workers. We make our corpus publicly available for further research.
References used
https://aclanthology.org/
rate research
Read More
Recent advances in neural machine translation (NMT) have pushed the quality of machine translation systems to the point where they are becoming widely adopted to build competitive systems. However, there is still a large number of languages that are
yet to reap the benefits of NMT. In this paper, we provide the first large-scale case study of the practical application of MT in the Turkic language family in order to realize the gains of NMT for Turkic languages under high-resource to extremely low-resource scenarios. In addition to presenting an extensive analysis that identifies the bottlenecks towards building competitive systems to ameliorate data scarcity, our study has several key contributions, including, i) a large parallel corpus covering 22 Turkic languages consisting of common public datasets in combination with new datasets of approximately 1.4 million parallel sentences, ii) bilingual baselines for 26 language pairs, iii) novel high-quality test sets in three different translation domains and iv) human evaluation scores. All models, scripts, and data will be released to the public.
Recent studies emphasize the need of document context in human evaluation of machine translations, but little research has been done on the impact of user interfaces on annotator productivity and the reliability of assessments. In this work, we compa
re human assessment data from the last two WMT evaluation campaigns collected via two different methods for document-level evaluation. Our analysis shows that a document-centric approach to evaluation where the annotator is presented with the entire document context on a screen leads to higher quality segment and document level assessments. It improves the correlation between segment and document scores and increases inter-annotator agreement for document scores but is considerably more time consuming for annotators.
This work introduces Itihasa, a large-scale translation dataset containing 93,000 pairs of Sanskrit shlokas and their English translations. The shlokas are extracted from two Indian epics viz., The Ramayana and The Mahabharata. We first describe the
motivation behind the curation of such a dataset and follow up with empirical analysis to bring out its nuances. We then benchmark the performance of standard translation models on this corpus and show that even state-of-the-art transformer architectures perform poorly, emphasizing the complexity of the dataset.
Recent development in NLP shows a strong trend towards refining pre-trained models with a domain-specific dataset. This is especially the case for response generation where emotion plays an important role. However, existing empathetic datasets remain
small, delaying research efforts in this area, for example, the development of emotion-aware chatbots. One main technical challenge has been the cost of manually annotating dialogues with the right emotion labels. In this paper, we describe a large-scale silver dataset consisting of 1M dialogues annotated with 32 fine-grained emotions, eight empathetic response intents, and the Neutral category. To achieve this goal, we have developed a novel data curation pipeline starting with a small seed of manually annotated data and eventually scaling it to a satisfactory size. We compare its quality against a state-of-the-art gold dataset using both offline experiments and visual validation methods. The resultant procedure can be used to create similar datasets in the same domain as well as in other domains.
Automatic image captioning has improved significantly over the last few years, but the problem is far from being solved, with state of the art models still often producing low quality captions when used in the wild. In this paper, we focus on the tas
k of Quality Estimation (QE) for image captions, which attempts to model the caption quality from a human perspective and *without* access to ground-truth references, so that it can be applied at prediction time to detect low-quality captions produced on *previously unseen images*. For this task, we develop a human evaluation process that collects coarse-grained caption annotations from crowdsourced users, which is then used to collect a large scale dataset spanning more than 600k caption quality ratings. We then carefully validate the quality of the collected ratings and establish baseline models for this new QE task. Finally, we further collect fine-grained caption quality annotations from trained raters, and use them to demonstrate that QE models trained over the coarse ratings can effectively detect and filter out low-quality image captions, thereby improving the user experience from captioning systems.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
