على الرغم من إظهار قيم واعدة للتطبيقات المصب، فإن توليد السؤال والإجابة معا يتم استكشافها. في هذه الورقة، نقدم مهمة جديدة تستهدف توليد زوج الإجابة على الأسئلة من الصور المرئية. لا يتطلب عدم توليد أزواج حول الإجابات المتنوعة فقط ولكن أيضا الحفاظ على الاتساق منهم. نحن ندرس نماذج جيل مختلفة لهذه المهمة واقتراح ثلاث نماذج: نموذج خط الأنابيب، النموذج المشترك، النموذج المتسلسل. نحن ندمج الاستدلال الاختلافي في هذه النماذج لتحقيق التنوع والاتساق. ونحن نقترح أيضا تقدير تمثيل المنطقة ومحاذاة الاهتمام لتحسين الاتساق أكثر. ونحن أخيرا وضع مقيم كقيط كمي للاتساق. نحن نقوم بالتحقق من نهجنا على معيارين، VQA2.0 و Visual-7W، من خلال تقييم التنوع والاتساق يدويا يدويا. تظهر النتائج التجريبية فعالية نماذجنا: يمكن أن تولد أزواج متنوعة أو متسقة. علاوة على ذلك، يمكن استخدام هذه المهمة لتحسين جيل السؤال المرئي والإجابة على السؤال المرئي.
Although showing promising values to downstream applications, generating question and answer together is under-explored. In this paper, we introduce a novel task that targets question-answer pair generation from visual images. It requires not only generating diverse question-answer pairs but also keeping the consistency of them. We study different generation paradigms for this task and propose three models: the pipeline model, the joint model, and the sequential model. We integrate variational inference into these models to achieve diversity and consistency. We also propose region representation scaling and attention alignment to improve the consistency further. We finally devise an evaluator as a quantitative metric for consistency. We validate our approach on two benchmarks, VQA2.0 and Visual-7w, by automatically and manually evaluating diversity and consistency. Experimental results show the effectiveness of our models: they can generate diverse or consistent pairs. Moreover, this task can be used to improve visual question generation and visual question answering.
References used
https://aclanthology.org/
Despite excellent performance on tasks such as question answering, Transformer-based architectures remain sensitive to syntactic and contextual ambiguities. Question Paraphrasing (QP) offers a promising solution as a means to augment existing dataset
Dual-Encoders is a promising mechanism for answer retrieval in question answering (QA) systems. Currently most conventional Dual-Encoders learn the semantic representations of questions and answers merely through matching score. Researchers proposed
Question answering (QA) models for reading comprehension have been demonstrated to exploit unintended dataset biases such as question--context lexical overlap. This hinders QA models from generalizing to under-represented samples such as questions wi
Story visualization is an underexplored task that falls at the intersection of many important research directions in both computer vision and natural language processing. In this task, given a series of natural language captions which compose a story
In question generation, the question produced has to be well-formed and meaningfully related to the answer serving as input. Neural generation methods have predominantly leveraged the distributional semantics of words as representations of meaning an