Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userQuality Estimation for Image Captions Based on Large-scale Human Evaluations
تقدير الجودة لمعاملات الصور بناء على تقييمات بشرية واسعة النطاق
268
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تحسنت تقسيم الصور التلقائي بشكل كبير خلال السنوات القليلة الماضية، لكن المشكلة بعيدة عن حلها، حيث لا تزال حالة من النماذج الفنية غالبا ما تنتج توضيحات منخفضة الجودة عند استخدامها في البرية. في هذه الورقة، نركز على مهمة تقدير الجودة (QE) للحصول على تعليق الصور، والتي تحاول طراز جودة التسمية التوضيحية من منظور إنساني و * بدون * الوصول إلى مراجع الحقيقة الأرضية، بحيث يمكن تطبيقها في وقت التنبؤ للكشف عن التسميات التوضيحية منخفضة الجودة المنتجة على * الصور غير المرئية سابقا *. بالنسبة لهذه المهمة، نقوم بتطوير عملية تقييم بشرية تقوم بجمع التعليقات التوضيحية الشعوية من المستخدمين من مستخدمي الجماعة الجماعية، والتي يتم استخدامها بعد ذلك لجمع مجموعة بيانات كبيرة الحجم تمتد أكثر من 600 كيلو بايت تصنيفات جودة التسمية التوضيحية. ثم صرفها بعناية جودة التصنيفات التي تم جمعها وإنشاء نماذج أساسية لهذه المهمة الجديدة في QE. أخيرا، سنقوم بزيادة جمع التعليقات التوضيحية ذات جودة التسمية التوضيحية الجميلة من الدراسات المدربة، واستخدامها لإظهار أن نماذج QE التي تم تدريبها على التصنيفات الخشنة يمكن أن تكتشف بشكل فعال وتصفية التسميات التوضيحية ذات الجودة المنخفضة الجودة، وبالتالي تحسين تجربة المستخدم من أنظمة التسمية التوضيحية.
Automatic image captioning has improved significantly over the last few years, but the problem is far from being solved, with state of the art models still often producing low quality captions when used in the wild. In this paper, we focus on the task of Quality Estimation (QE) for image captions, which attempts to model the caption quality from a human perspective and *without* access to ground-truth references, so that it can be applied at prediction time to detect low-quality captions produced on *previously unseen images*. For this task, we develop a human evaluation process that collects coarse-grained caption annotations from crowdsourced users, which is then used to collect a large scale dataset spanning more than 600k caption quality ratings. We then carefully validate the quality of the collected ratings and establish baseline models for this new QE task. Finally, we further collect fine-grained caption quality annotations from trained raters, and use them to demonstrate that QE models trained over the coarse ratings can effectively detect and filter out low-quality image captions, thereby improving the user experience from captioning systems.
References used
https://aclanthology.org/
rate research
Read More
Abstract Human evaluation of modern high-quality machine translation systems is a difficult problem, and there is increasing evidence that inadequate evaluation procedures can lead to erroneous conclusions. While there has been considerable research
on human evaluation, the field still lacks a commonly accepted standard procedure. As a step toward this goal, we propose an evaluation methodology grounded in explicit error analysis, based on the Multidimensional Quality Metrics (MQM) framework. We carry out the largest MQM research study to date, scoring the outputs of top systems from the WMT 2020 shared task in two language pairs using annotations provided by professional translators with access to full document context. We analyze the resulting data extensively, finding among other results a substantially different ranking of evaluated systems from the one established by the WMT crowd workers, exhibiting a clear preference for human over machine output. Surprisingly, we also find that automatic metrics based on pre-trained embeddings can outperform human crowd workers. We make our corpus publicly available for further research.
Recent development in NLP shows a strong trend towards refining pre-trained models with a domain-specific dataset. This is especially the case for response generation where emotion plays an important role. However, existing empathetic datasets remain
small, delaying research efforts in this area, for example, the development of emotion-aware chatbots. One main technical challenge has been the cost of manually annotating dialogues with the right emotion labels. In this paper, we describe a large-scale silver dataset consisting of 1M dialogues annotated with 32 fine-grained emotions, eight empathetic response intents, and the Neutral category. To achieve this goal, we have developed a novel data curation pipeline starting with a small seed of manually annotated data and eventually scaling it to a satisfactory size. We compare its quality against a state-of-the-art gold dataset using both offline experiments and visual validation methods. The resultant procedure can be used to create similar datasets in the same domain as well as in other domains.
Large-scale language models such as GPT-3 are excellent few-shot learners, allowing them to be controlled via natural text prompts. Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and i
nference scalability. This paper proposes a novel data augmentation technique that leverages large-scale language models to generate realistic text samples from a mixture of real samples. We also propose utilizing soft-labels predicted by the language models, effectively distilling knowledge from the large-scale language models and creating textual perturbations simultaneously. We perform data augmentation experiments on diverse classification tasks and show that our method hugely outperforms existing text augmentation methods. We also conduct experiments on our newly proposed benchmark to show that the augmentation effect is not only attributed to memorization. Further ablation studies and a qualitative analysis provide more insights into our approach.
This work introduces Itihasa, a large-scale translation dataset containing 93,000 pairs of Sanskrit shlokas and their English translations. The shlokas are extracted from two Indian epics viz., The Ramayana and The Mahabharata. We first describe the
motivation behind the curation of such a dataset and follow up with empirical analysis to bring out its nuances. We then benchmark the performance of standard translation models on this corpus and show that even state-of-the-art transformer architectures perform poorly, emphasizing the complexity of the dataset.
This paper introduces MediaSum, a large-scale media interview dataset consisting of 463.6K transcripts with abstractive summaries. To create this dataset, we collect interview transcripts from NPR and CNN and employ the overview and topic description
s as summaries. Compared with existing public corpora for dialogue summarization, our dataset is an order of magnitude larger and contains complex multi-party conversations from multiple domains. We conduct statistical analysis to demonstrate the unique positional bias exhibited in the transcripts of televised and radioed interviews. We also show that MediaSum can be used in transfer learning to improve a model's performance on other dialogue summarization tasks.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
