Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userEnhancing Visual Dialog Questioner with Entity-based Strategy Learning and Augmented Guesser
تعزيز سؤال الحوار المرئي مع تعلم الإستراتيجية القائمة على الكيان والتخصيص المعزز
161
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
بالنظر إلى أهمية بناء أسئلة حوار مرئي جيد (VD)، يدرس العديد من الباحثين موضوعا بموجب إعداد لعبة تخمين صورة Q-Bot-A-Bot، حيث يحتاج السائل إلى رفع سلسلة من الأسئلة لجمع معلومات غير معلنة صورة. على الرغم من التقدم المحرز في التعلم الإشراف (SL) وتعزيز التعزيز (RL)، فإن القضايا لا تزال موجودة. أولا، لا توفر الأساليب السابقة إرشادات صريحة وفعالة للمسائل لتوليد أسئلة ذات صلة وبصراحية. ثانيا، يتم إعاقة تأثير RL بواسطة مكون غير كفء، أي خيال، الذي يصنع تنبؤات الصور بناء على الحوار التي تم إنشاؤها وتعيين المكافآت وفقا لذلك. لتعزيز مسائل VD: 1) نقترح اسئلة محسنة للكيان (REEQ) التي تولد أسئلة تحت إشراف الكيانات ذات الصلة وتعلم استراتيجية الاستجواب القائمة على الكيان من الحوار البشرية؛ 2) نقترح خيال معدني قوي ويتم تحسينه ل VD خاصة. تظهر النتائج التجريبية على مجموعة بيانات V1.0 Visdial أن نهجنا يحقق أداء حديثة على كل من مهمة تخمين الصورة وتنوع السؤال. تتحقق الدراسة البشرية مزيدا من أن نموذجنا يولد أسئلة أكثر متعلقة بصريا ومفيدة ومتماسكة.
Considering the importance of building a good Visual Dialog (VD) Questioner, many researchers study the topic under a Q-Bot-A-Bot image-guessing game setting, where the Questioner needs to raise a series of questions to collect information of an undisclosed image. Despite progress has been made in Supervised Learning (SL) and Reinforcement Learning (RL), issues still exist. Firstly, previous methods do not provide explicit and effective guidance for Questioner to generate visually related and informative questions. Secondly, the effect of RL is hampered by an incompetent component, i.e., the Guesser, who makes image predictions based on the generated dialogs and assigns rewards accordingly. To enhance VD Questioner: 1) we propose a Related entity enhanced Questioner (ReeQ) that generates questions under the guidance of related entities and learns entity-based questioning strategy from human dialogs; 2) we propose an Augmented Guesser that is strong and is optimized for VD especially. Experimental results on the VisDial v1.0 dataset show that our approach achieves state-of-the-art performance on both image-guessing task and question diversity. Human study further verifies that our model generates more visually related, informative and coherent questions.
References used
https://aclanthology.org/
rate research
Read More
Visual dialog is challenging since it needs to answer a series of coherent questions based on understanding the visual environment. How to ground related visual objects is one of the key problems. Previous studies utilize the question and history to
attend to the image and achieve satisfactory performance, while these methods are not sufficient to locate related visual objects without any guidance. The inappropriate grounding of visual objects prohibits the performance of visual dialog models. In this paper, we propose a novel approach to Learn to Ground visual objects for visual dialog, which employs a novel visual objects grounding mechanism where both prior and posterior distributions over visual objects are used to facilitate visual objects grounding. Specifically, a posterior distribution over visual objects is inferred from both context (history and questions) and answers, and it ensures the appropriate grounding of visual objects during the training process. Meanwhile, a prior distribution, which is inferred from context only, is used to approximate the posterior distribution so that appropriate visual objects can be grounding even without answers during the inference process. Experimental results on the VisDial v0.9 and v1.0 datasets demonstrate that our approach improves the previous strong models in both generative and discriminative settings by a significant margin.
Visual dialog is a task of answering a sequence of questions grounded in an image using the previous dialog history as context. In this paper, we study how to address two fundamental challenges for this task: (1) reasoning over underlying semantic st
ructures among dialog rounds and (2) identifying several appropriate answers to the given question. To address these challenges, we propose a Sparse Graph Learning (SGL) method to formulate visual dialog as a graph structure learning task. SGL infers inherently sparse dialog structures by incorporating binary and score edges and leveraging a new structural loss function. Next, we introduce a Knowledge Transfer (KT) method that extracts the answer predictions from the teacher model and uses them as pseudo labels. We propose KT to remedy the shortcomings of single ground-truth labels, which severely limit the ability of a model to obtain multiple reasonable answers. As a result, our proposed model significantly improves reasoning capability compared to baseline methods and outperforms the state-of-the-art approaches on the VisDial v1.0 dataset. The source code is available at https://github.com/gicheonkang/SGLKT-VisDial.
Screenplay summarization is the task of extracting informative scenes from a screenplay. The screenplay contains turning point (TP) events that change the story direction and thus define the story structure decisively. Accordingly, this task can be d
efined as the TP identification task. We suggest using dialogue information, one attribute of screenplays, motivated by previous work that discovered that TPs have a relation with dialogues appearing in screenplays. To teach a model this characteristic, we add a dialogue feature to the input embedding. Moreover, in an attempt to improve the model architecture of previous studies, we replace LSTM with Transformer. We observed that the model can better identify TPs in a screenplay by using dialogue information and that a model adopting Transformer outperforms LSTM-based models.
Various machine learning tasks can benefit from access to external information of different modalities, such as text and images. Recent work has focused on learning architectures with large memories capable of storing this knowledge. We propose augme
nting generative Transformer neural networks with KNN-based Information Fetching (KIF) modules. Each KIF module learns a read operation to access fixed external knowledge. We apply these modules to generative dialog modeling, a challenging task where information must be flexibly retrieved and incorporated to maintain the topic and flow of conversation. We demonstrate the effectiveness of our approach by identifying relevant knowledge required for knowledgeable but engaging dialog from Wikipedia, images, and human-written dialog utterances, and show that leveraging this retrieved information improves model performance, measured by automatic and human evaluation.
For each goal-oriented dialog task of interest, large amounts of data need to be collected for end-to-end learning of a neural dialog system. Collecting that data is a costly and time-consuming process. Instead, we show that we can use only a small a
mount of data, supplemented with data from a related dialog task. Naively learning from related data fails to improve performance as the related data can be inconsistent with the target task. We describe a meta-learning based method that selectively learns from the related dialog task data. Our approach leads to significant accuracy improvements in an example dialog task.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
