Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userSelf-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents
التعلم المتعاقل الذي يشرف على نفسه لتنبؤ برضيا المستخدم فعال في وكلاء المحادثة
140
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
رضا المستخدمين على مستوى الدوران هو أحد أهم مقاييس الأداء لعوامل المحادثة. يمكن استخدامه لمراقبة أداء الوكيل وتوفير رؤى حول تجارب المستخدم المعيبة. في حين أن التعلم العميق المنتهي في النهاية قد أظهر نتائج واعدة، فإن الوصول إلى عدد كبير من العينات المشروح الموثوقة التي تتطلبها هذه الطرق تظل تحديا. في نظام محادثة واسعة النطاق، يوجد عدد متزايد من المهارات المتقدمة حديثا، مما يجعل عملية جمع البيانات التقليدية والشروحية وعملية النمذجة غير عملي بسبب تكاليف التوضيحية المطلوبة وأوقات التحول. في هذه الورقة، نقترح اقتراح نهج تعليمي بسيط للإشراف على أن يهدف إلى مجموعة من البيانات غير المسبقة لتعلم تفاعلات وكيل المستخدم. نظهر أن النماذج المدربة مسبقا باستخدام الهدف الأكثر إشرا للإشراف قابلة للتحويل إلى تنبؤ رضا المستخدمين. بالإضافة إلى ذلك، نقترح نقه نهج لتعلم تحويل القليل من الرواية يضمن نقل أفضل لأحجام عينة صغيرة جدا. لا تتطلب الطريقة القليلة المقترحة أي عملية تحسين الحلقة الداخلية وهي قابلة للتحجيم إلى مجموعات البيانات الكبيرة جدا والنماذج المعقدة. بناء على تجاربنا باستخدام بيانات حقيقية من نظام تجاري واسع النطاق، فإن النهج المقترح قادر على تقليل العدد المطلوب بشكل كبير، مع تحسين التعميم بشأن المهارات غير المرئية.
Turn-level user satisfaction is one of the most important performance metrics for conversational agents. It can be used to monitor the agent's performance and provide insights about defective user experiences. While end-to-end deep learning has shown promising results, having access to a large number of reliable annotated samples required by these methods remains challenging. In a large-scale conversational system, there is a growing number of newly developed skills, making the traditional data collection, annotation, and modeling process impractical due to the required annotation costs and the turnaround times. In this paper, we suggest a self-supervised contrastive learning approach that leverages the pool of unlabeled data to learn user-agent interactions. We show that the pre-trained models using the self-supervised objective are transferable to the user satisfaction prediction. In addition, we propose a novel few-shot transfer learning approach that ensures better transferability for very small sample sizes. The suggested few-shot method does not require any inner loop optimization process and is scalable to very large datasets and complex models. Based on our experiments using real data from a large-scale commercial system, the suggested approach is able to significantly reduce the required number of annotations, while improving the generalization on unseen skills.
References used
https://aclanthology.org/
rate research
Read More
Unlike well-structured text, such as news reports and encyclopedia articles, dialogue content often comes from two or more interlocutors, exchanging information with each other. In such a scenario, the topic of a conversation can vary upon progressio
n and the key information for a certain topic is often scattered across multiple utterances of different speakers, which poses challenges to abstractly summarize dialogues. To capture the various topic information of a conversation and outline salient facts for the captured topics, this work proposes two topic-aware contrastive learning objectives, namely coherence detection and sub-summary generation objectives, which are expected to implicitly model the topic change and handle information scattering challenges for the dialogue summarization task. The proposed contrastive objectives are framed as auxiliary tasks for the primary dialogue summarization task, united via an alternative parameter updating strategy. Extensive experiments on benchmark datasets demonstrate that the proposed simple method significantly outperforms strong baselines and achieves new state-of-the-art performance. The code and trained models are publicly available via .
We study generating abstractive summaries that are faithful and factually consistent with the given articles. A novel contrastive learning formulation is presented, which leverages both reference summaries, as positive training data, and automaticall
y generated erroneous summaries, as negative training data, to train summarization systems that are better at distinguishing between them. We further design four types of strategies for creating negative samples, to resemble errors made commonly by two state-of-the-art models, BART and PEGASUS, found in our new human annotations of summary errors. Experiments on XSum and CNN/Daily Mail show that our contrastive learning framework is robust across datasets and models. It consistently produces more factual summaries than strong comparisons with post error correction, entailment-based reranking, and unlikelihood training, according to QA-based factuality evaluation. Human judges echo the observation and find that our model summaries correct more errors.
In recent years, world business in online discussions and opinion sharing on social media is booming. Re-entry prediction task is thus proposed to help people keep track of the discussions which they wish to continue. Nevertheless, existing works onl
y focus on exploiting chatting history and context information, and ignore the potential useful learning signals underlying conversation data, such as conversation thread patterns and repeated engagement of target users, which help better understand the behavior of target users in conversations. In this paper, we propose three interesting and well-founded auxiliary tasks, namely, Spread Pattern, Repeated Target user, and Turn Authorship, as the self-supervised signals for re-entry prediction. These auxiliary tasks are trained together with the main task in a multi-task manner. Experimental results on two datasets newly collected from Twitter and Reddit show that our method outperforms the previous state-of-the-arts with fewer parameters and faster convergence. Extensive experiments and analysis show the effectiveness of our proposed models and also point out some key ideas in designing self-supervised tasks.
Multi-party dialogue machine reading comprehension (MRC) brings tremendous challenge since it involves multiple speakers at one dialogue, resulting in intricate speaker information flows and noisy dialogue contexts. To alleviate such difficulties, pr
evious models focus on how to incorporate these information using complex graph-based modules and additional manually labeled data, which is usually rare in real scenarios. In this paper, we design two labour-free self- and pseudo-self-supervised prediction tasks on speaker and key-utterance to implicitly model the speaker information flows, and capture salient clues in a long dialogue. Experimental results on two benchmark datasets have justified the effectiveness of our method over competitive baselines and current state-of-the-art models.
Conversational agents trained on large unlabeled corpora of human interactions will learn patterns and mimic behaviors therein, which include offensive or otherwise toxic behavior. We introduce a new human-and-model-in-the-loop framework for evaluati
ng the toxicity of such models, and compare a variety of existing methods in both the cases of non-adversarial and adversarial users that expose their weaknesses. We then go on to propose two novel methods for safe conversational agents, by either training on data from our new human-and-model-in-the-loop framework in a two-stage system, or ''baking-in'' safety to the generative model itself. We find our new techniques are (i) safer than existing models; while (ii) maintaining usability metrics such as engagingness relative to state-of-the-art chatbots. In contrast, we expose serious safety issues in existing standard systems like GPT2, DialoGPT, and BlenderBot.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
