Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userHow ``open'' are the conversations with open-domain chatbots? A proposal for Speech Event based evaluation
كيف "فتح" هي المحادثات التي تحتوي على chatbots المجال المفتوح؟اقتراح للتقييم القائم على أحداث الكلام
304
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
من المفترض أن يتحدث Chatbots المجال المفتوح بحرية مع البشر دون أن يقتصر على موضوع أو مهمة أو مجال. ومع ذلك، فإن حدود و / أو محتويات المحادثات المفتوحة ليست واضحة. لتوضيح حدود الانفتاح "، نقوم بإجراء دراستين: أولا، نقوم بتصنيف أنواع أحداث الكلام" واجهتها في مجموعة بيانات تقييم ChatBot (أي مينا من Google) وتجد أن هذه المحادثات تغطي بشكل أساسي الكلام الصغير بشكل أساسي "الفئة واستبعاد وفئات أحداث الكلام الأخرى التي تواجهها في الحياة البشرية الحقيقية البشرية. ثانيا، نقوم بإجراء دراسة تجريبية صغيرة على نطاق واسع لتوليد محادثات عبر الإنترنت تغطي مجموعة واسعة من فئات أحداث الكلام بين إطارين مقابل رجل بشري وحديث من شاتبوت (I.E.، خلاط على Facebook). يشير التقييم البشري لهذه المحادثات الناتجة إلى تفضيل للمحادثات البشرية، لأن محادثات الإنسان التي تشاتبوت تفتقر إلى التماسك في معظم فئات أحداث الكلام. بناء على هذه النتائج، نقترح (أ) استخدام مصطلح الحديث الصغير "بدلا من المجال المفتوح" للاتحاد الحالي الذي لا يفتح "من حيث قدرات المحادثة بعد، و (ب) مراجعة أساليب التقييم لاختبار محادثات Chatbot ضد أحداث الكلام الأخرى.
Open-domain chatbots are supposed to converse freely with humans without being restricted to a topic, task or domain. However, the boundaries and/or contents of open-domain conversations are not clear. To clarify the boundaries of openness'', we conduct two studies: First, we classify the types of speech events'' encountered in a chatbot evaluation data set (i.e., Meena by Google) and find that these conversations mainly cover the small talk'' category and exclude the other speech event categories encountered in real life human-human communication. Second, we conduct a small-scale pilot study to generate online conversations covering a wider range of speech event categories between two humans vs. a human and a state-of-the-art chatbot (i.e., Blender by Facebook). A human evaluation of these generated conversations indicates a preference for human-human conversations, since the human-chatbot conversations lack coherence in most speech event categories. Based on these results, we suggest (a) using the term small talk'' instead of open-domain'' for the current chatbots which are not that open'' in terms of conversational abilities yet, and (b) revising the evaluation methods to test the chatbot conversations against other speech events.
References used
https://aclanthology.org/
rate research
Read More
Open-domain question answering aims at locating the answers to user-generated questions in massive collections of documents. Retriever-readers and knowledge graph approaches are two big families of solutions to this task. A retriever-reader first app
lies information retrieval techniques to locate a few passages that are likely to be relevant, and then feeds the retrieved text to a neural network reader to extract the answer. Alternatively, knowledge graphs can be constructed and queried to answer users' questions. We propose an algorithm with a novel reader-retriever design that differs from both families. Our reader-retriever first uses an offline reader to read the corpus and generate collections of all answerable questions associated with their answers, and then uses an online retriever to respond to user queries by searching the pre-constructed question spaces for answers that are most likely to be asked in the given way. We further combine one retriever-reader and two reader-retrievers into a hybrid model called R6 for the best performance. Experiments with two large-scale public datasets show that R6 achieves state-of-the-art accuracy.
An overarching goal of natural language processing is to enable machines to communicate seamlessly with humans. However, natural language can be ambiguous or unclear. In cases of uncertainty, humans engage in an interactive process known as repair: a
sking questions and seeking clarification until their uncertainty is resolved. We propose a framework for building a visually grounded question-asking model capable of producing polar (yes-no) clarification questions to resolve misunderstandings in dialogue. Our model uses an expected information gain objective to derive informative questions from an off-the-shelf image captioner without requiring any supervised question-answer data. We demonstrate our model's ability to pose questions that improve communicative success in a goal-oriented 20 questions game with synthetic and human answerers.
Abstract Aspect-based summarization is the task of generating focused summaries based on specific points of interest. Such summaries aid efficient analysis of text, such as quickly understanding reviews or opinions from different angles. However, due
to large differences in the type of aspects for different domains (e.g., sentiment, product features), the development of previous models has tended to be domain-specific. In this paper, we propose WikiAsp,1 a large-scale dataset for multi-domain aspect- based summarization that attempts to spur research in the direction of open-domain aspect-based summarization. Specifically, we build the dataset using Wikipedia articles from 20 different domains, using the section titles and boundaries of each article as a proxy for aspect annotation. We propose several straightforward baseline models for this task and conduct experiments on the dataset. Results highlight key challenges that existing summarization models face in this setting, such as proper pronoun handling of quoted sources and consistent explanation of time-sensitive events.
Abstract ⚠ This paper contains prompts and model outputs that are offensive in nature. When trained on large, unfiltered crawls from the Internet, language models pick up and reproduce all kinds of undesirable biases that can be found in the data: Th
ey often generate racist, sexist, violent, or otherwise toxic language. As large models require millions of training examples to achieve good performance, it is difficult to completely prevent them from being exposed to such content. In this paper, we first demonstrate a surprising finding: Pretrained language models recognize, to a considerable degree, their undesirable biases and the toxicity of the content they produce. We refer to this capability as self-diagnosis. Based on this finding, we then propose a decoding algorithm that, given only a textual description of the undesired behavior, reduces the probability of a language model producing problematic text. We refer to this approach as self-debiasing. Self-debiasing does not rely on manually curated word lists, nor does it require any training data or changes to the model's parameters. While we by no means eliminate the issue of language models generating biased text, we believe our approach to be an important step in this direction.1
We introduce a new dataset for Question Rewriting in Conversational Context (QReCC), which contains 14K conversations with 80K question-answer pairs. The task in QReCC is to find answers to conversational questions within a collection of 10M web page
s (split into 54M passages). Answers to questions in the same conversation may be distributed across several web pages. QReCC provides annotations that allow us to train and evaluate individual subtasks of question rewriting, passage retrieval and reading comprehension required for the end-to-end conversational question answering (QA) task. We report the effectiveness of a strong baseline approach that combines the state-of-the-art model for question rewriting, and competitive models for open-domain QA. Our results set the first baseline for the QReCC dataset with F1 of 19.10, compared to the human upper bound of 75.45, indicating the difficulty of the setup and a large room for improvement.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
