Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userGenerating Negative Samples by Manipulating Golden Responses for Unsupervised Learning of a Response Evaluation Model
توليد العينات السلبية عن طريق معالجة الاستجابات الذهبية للتعلم غير المعدل لنموذج تقييم الاستجابة
225
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تقييم جودة الردود الناتجة عن أنظمة محادثة المجال المفتوحة هي مهمة صعبة. هذا جزئيا لأنه يمكن أن يكون هناك العديد من الردود المناسبة لتاريخ حوار معين. غالبا ما تفشل المقاييس المرجعية التي تعتمد على مقارنات إلى مجموعة من الاستجابات الصحيحة المعروفة في حساب هذا التنوع، وبالتالي ربط سيئة مع الحكم البشري. لمعالجة هذه المشكلة، قام الباحثون بالتحقيق في إمكانية تقييم جودة الاستجابة دون استخدام مجموعة من الردود الصحيحة المعروفة. أظهر روبر أنه يمكن إجراء نموذج تقييم الاستجابة التلقائي باستخدام التعلم غير المزعوم لمهمة التنبؤ بالكلام التالي (NUP). بالنسبة للتعلم غير المقترح لهذا النموذج، نقترح طريقة التلاعب بالاستجابة الذهبية لإنشاء استجابة سلبية جديدة تم تصميمها لتكون غير مناسب في السياق مع الحفاظ على التشابه العالي مع الاستجابة الذهبية الأصلية. نجد، من تجاربنا في مجموعات البيانات الإنجليزية، التي تستخدم العينات السلبية التي تم إنشاؤها بواسطة طريقتنا إلى جانب العينات السلبية العشوائية يمكن أن تزيد من ارتباط النموذج بالتقييمات البشرية. عملية توليد هذه العينات السلبية مؤتمتة ولا تعتمد على شرح الإنسان.
Evaluating the quality of responses generated by open-domain conversation systems is a challenging task. This is partly because there can be multiple appropriate responses to a given dialogue history. Reference-based metrics that rely on comparisons to a set of known correct responses often fail to account for this variety, and consequently correlate poorly with human judgment. To address this problem, researchers have investigated the possibility of assessing response quality without using a set of known correct responses. RUBER demonstrated that an automatic response evaluation model could be made using unsupervised learning for the next-utterance prediction (NUP) task. For the unsupervised learning of such model, we propose a method of manipulating a golden response to create a new negative response that is designed to be inappropriate within the context while maintaining high similarity with the original golden response. We find, from our experiments on English datasets, that using the negative samples generated by our method alongside random negative samples can increase the model's correlation with human evaluations. The process of generating such negative samples is automated and does not rely on human annotation.
References used
https://aclanthology.org/
rate research
Read More
Humans make appropriate responses not only based on previous dialogue utterances but also on implicit background knowledge such as common sense. Although neural response generation models seem to produce human-like responses, they are mostly end-to-e
nd and not generating intermediate grounds between a dialogue history and responses. This work aims to study if and how we can train an RG model that talks with itself to generate implicit knowledge before making responses. We further investigate can such models identify when to generate implicit background knowledge and when it is not necessary. Experimental results show that compared with models that directly generate responses given a dialogue history, self-talk models produce better-quality responses according to human evaluation on grammaticality, coherence, and engagingness. And models that are trained to identify when to self-talk further improves the response quality. Analysis on generated implicit knowledge shows that models mostly use the knowledge appropriately in the responses.
Empathy is a complex cognitive ability based on the reasoning of others' affective states. In order to better understand others and express stronger empathy in dialogues, we argue that two issues must be tackled at the same time: (i) identifying whic
h word is the cause for the other's emotion from his or her utterance and (ii) reflecting those specific words in the response generation. However, previous approaches for recognizing emotion cause words in text require sub-utterance level annotations, which can be demanding. Taking inspiration from social cognition, we leverage a generative estimator to infer emotion cause words from utterances with no word-level label. Also, we introduce a novel method based on pragmatics to make dialogue models focus on targeted words in the input during generation. Our method is applicable to any dialogue models with no additional training on the fly. We show our approach improves multiple best-performing dialogue agents on generating more focused empathetic responses in terms of both automatic and human evaluation.
Selectional Preference (SP) captures the tendency of a word to semantically select other words to be in direct syntactic relation with it, and thus informs us about syntactic word configurations that are meaningful. Therefore SP is a valuable resourc
e for Natural Language Processing (NLP) systems and for semanticists. Learning SP has generally been seen as a supervised task, because it requires a parsed corpus as a source of syntactically related word pairs. In this paper we show that simple distributional analysis can learn a good amount of SP without the need for an annotated corpus. We extend the general word embedding technique with directional word context windows giving word representations that better capture syntagmatic relations. We test on the SP-10K dataset and demonstrate that syntagmatic embeddings outperform the paradigmatic embeddings. We also evaluate supervised version of these embeddings and show that unsupervised syntagmatic embeddings can be as good as supervised embeddings. We also make available the source code of our implementation.
Multi-turn response selection models have recently shown comparable performance to humans in several benchmark datasets. However, in the real environment, these models often have weaknesses, such as making incorrect predictions based heavily on super
ficial patterns without a comprehensive understanding of the context. For example, these models often give a high score to the wrong response candidate containing several keywords related to the context but using the inconsistent tense. In this study, we analyze the weaknesses of the open-domain Korean Multi-turn response selection models and publish an adversarial dataset to evaluate these weaknesses. We also suggest a strategy to build a robust model in this adversarial environment.
Motivated by suggested question generation in conversational news recommendation systems, we propose a model for generating question-answer pairs (QA pairs) with self-contained, summary-centric questions and length-constrained, article-summarizing an
swers. We begin by collecting a new dataset of news articles with questions as titles and pairing them with summaries of varying length. This dataset is used to learn a QA pair generation model producing summaries as answers that balance brevity with sufficiency jointly with their corresponding questions. We then reinforce the QA pair generation process with a differentiable reward function to mitigate exposure bias, a common problem in natural language generation. Both automatic metrics and human evaluation demonstrate these QA pairs successfully capture the central gists of the articles and achieve high answer accuracy.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
