نقوم بإجراء تقييم بشري في الأوراق التي تقدم العمل على توليد اللغة الطبيعية الإبداعية التي تم نشرها في INLG 2020 و ICCC 2020. أكثر طريقة التقييم البشرية النموذجية هي مسح كبير الحجم، وعادة ما تكون على نطاق 5 نقاط، في حين وجود العديد من الأساليب الأقل شيوعا أخرى.المعلمات الأكثر شيوعا هي معنى، صحة النحوية، الجدة والأهمية والقيمة العاطفية، من بين العديد من الآخرين.تشمل المبادئ التوجيهية الخاصة بنا للتقييم المستقبلي بوضوح هدف النظام التوليدي، وطرح أسئلة كملموسة قدر الإمكان، واختبار إعداد التقييم، باستخدام إعدادات تقييم متعددة متعددة، وإبلاغ عملية التقييم بأكملها والتحيزات المحتملة بوضوح، وأخيرا تحليل نتائج التقييم في النهايةبطريقة أكثر عمقا من إبلاغ الإحصاءات الأكثر نموذجية.
We survey human evaluation in papers presenting work on creative natural language generation that have been published in INLG 2020 and ICCC 2020. The most typical human evaluation method is a scaled survey, typically on a 5 point scale, while many other less common methods exist. The most commonly evaluated parameters are meaning, syntactic correctness, novelty, relevance and emotional value, among many others. Our guidelines for future evaluation include clearly defining the goal of the generative system, asking questions as concrete as possible, testing the evaluation setup, using multiple different evaluation setups, reporting the entire evaluation process and potential biases clearly, and finally analyzing the evaluation results in a more profound way than merely reporting the most typical statistics.
References used
https://aclanthology.org/
This paper aims at providing a comprehensive overview of recent developments in dialogue state tracking (DST) for task-oriented conversational systems. We introduce the task, the main datasets that have been exploited as well as their evaluation metr
Translation Studies and more specifically, its subfield Descriptive Translation Studies [Holmes 1988/2000] is, according to many scholars [Gambier, 2009; Nenopoulou, 2007; Munday, 2001/2008; Hermans, 1999; Snell-Hornby et al., 1994 e.t.c], a highly i
This paper provides a quick overview of possible methods how to detect that reference translations were actually created by post-editing an MT system. Two methods based on automatic metrics are presented: BLEU difference between the suspected MT and
This paper reviews and summarizes human evaluation practices described in 97 style transfer papers with respect to three main evaluation aspects: style transfer, meaning preservation, and fluency. In principle, evaluations by human raters should be t
We outline the Great Misalignment Problem in natural language processing research, this means simply that the problem definition is not in line with the method proposed and the human evaluation is not in line with the definition nor the method. We st