Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userWhat Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts
ماذا حصل؟شرح جودة الحوار الشامل من خلال تأثيرات على مستوى الكلام
168
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
غالبا ما يتطلب تحسين تجربة المستخدم لنظام الحوار جهدا مكثفا مطورا مكثفا لقراءة سجلات المحادثة، وتشغيل التحليلات الإحصائية، والأهمية النسبية لأوجه القصور النسبية.تقدم هذه الورقة نهجا جديدا للتحليل الآلي لسجلات المحادثة التي تتعلم العلاقة بين تفاعلات نظام المستخدم وجودة الحوار الشاملة.على عكس العمل السابق على التنبؤ بجودة الكلام على مستوى الكلام، يتعلم نهجنا تأثير كل تفاعل من تصنيف المستخدمين العام دون إشراف على مستوى الكلام، مما يسمح باستنتاجات النماذج الناتجة عن الاستمتاع على أساس الأدلة التجريبية وتكلفة منخفضة.يحدد نموذجنا التفاعلات التي لها علاقة قوية بجودة الحوار الشاملة في إعداد chatbot.تشير التجارب إلى أن التحليل الآلي من طرازنا يوافق على أحكام الخبراء، مما يجعل هذا العمل الأول من يوضح أن هذا التعلم الإشرافه ضعيف في التنبؤ بجودة الكلام هو قابلة للتحقيق بشدة.
Improving user experience of a dialogue system often requires intensive developer effort to read conversation logs, run statistical analyses, and intuit the relative importance of system shortcomings. This paper presents a novel approach to automated analysis of conversation logs that learns the relationship between user-system interactions and overall dialogue quality. Unlike prior work on utterance-level quality prediction, our approach learns the impact of each interaction from the overall user rating without utterance-level annotation, allowing resultant model conclusions to be derived on the basis of empirical evidence and at low cost. Our model identifies interactions that have a strong correlation with the overall dialogue quality in a chatbot setting. Experiments show that the automated analysis from our model agrees with expert judgments, making this work the first to show that such weakly-supervised learning of utterance-level quality prediction is highly achievable.
References used
https://aclanthology.org/
rate research
Read More
Recent task-oriented dialogue systems learn a model from annotated dialogues, and such dialogues are in turn collected and annotated so that they are consistent with certain domain knowledge. However, in real scenarios, domain knowledge is subject to
frequent changes, and initial training dialogues may soon become obsolete, resulting in a significant decrease in the model performance. In this paper, we investigate the relationship between training dialogues and domain knowledge, and propose Dialogue Domain Adaptation, a methodology aiming at adapting initial training dialogues to changes intervened in the domain knowledge. We focus on slot-value changes (e.g., when new slot values are available to describe domain entities) and define an experimental setting for dialogue domain adaptation. First, we show that current state-of-the-art models for dialogue state tracking are still poorly robust to slot-value changes of the domain knowledge. Then, we compare different domain adaptation strategies, showing that simple techniques are effective to reduce the gap between training dialogues and domain knowledge.
Identifying relevant knowledge to be used in conversational systems that are grounded in long documents is critical to effective response generation. We introduce a knowledge identification model that leverages the document structure to provide dialo
gue-contextualized passage encodings and better locate knowledge relevant to the conversation. An auxiliary loss captures the history of dialogue-document connections. We demonstrate the effectiveness of our model on two document-grounded conversational datasets and provide analyses showing generalization to unseen documents and long dialogue contexts.
This study proposes an utterance position-aware approach for a neural network-based dialogue act recognition (DAR) model, which incorporates positional encoding for utterance's absolute or relative position. The proposed approach is inspired by the o
bservation that some dialogue acts have tendencies of occurrence positions. The evaluations on the Switchboard corpus show that the proposed positional encoding of utterances statistically significantly improves the performance of DAR.
Word embeddings capture semantic meaning of individual words. How to bridge word-level linguistic knowledge with sentence-level language representation is an open problem. This paper examines whether sentence-level representations can be achieved by
building a custom sentence database focusing on one aspect of a sentence's meaning. Our three separate semantic aspects are whether the sentence: (1) communicates a causal relationship, (2) indicates that two things are correlated with each other, and (3) expresses information or knowledge. The three classifiers provide epistemic information about a sentence's content.
Knowledge Grounded Conversation Models are usually based on a selection/retrieval module and a generation module, trained separately or simultaneously, with or without having access to a gold' knowledge option. With the introduction of large pre-trai
ned generative models, the selection and generation part have become more and more entangled, shifting the focus towards enhancing knowledge incorporation (from multiple sources) instead of trying to pick the best knowledge option. These approaches however depend on knowledge labels and/or a separate dense retriever for their best performance. In this work we study the unsupervised selection abilities of pre-trained generative models (e.g. BART) and show that by adding a score-and-aggregate module between encoder and decoder, they are capable of learning to pick the proper knowledge through minimising the language modelling loss (i.e. without having access to knowledge labels). Trained as such, our model - K-Mine - shows competitive selection and generation performance against models that benefit from knowledge labels and/or separate dense retriever.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
