Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userCan Question Generation Debias Question Answering Models? A Case Study on Question--Context Lexical Overlap
يمكن أن تشكك جيل ديبيا سؤال الإجابة النماذج؟دراسة حالة عن السؤال - التداخل المعجمي السياق
374
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تم إظهار نماذج الإجابة على الأسئلة (QA) للحصول على فهم القراءة لاستغلال تحيزات محددات غير مقصودة مثل السؤال - التداخل المعجمي السياق. هذا يعيق نماذج ضمان الجودة من المعمم إلى العينات الممثلة تمثيلا مثل الأسئلة ذات التداخل المعجمي المنخفض. يمكن أن يكون جيل السؤال (QG)، وهي طريقة لتعزيز مجموعات بيانات QA، حل تدهور الأداء إذا كان QG يمكن أن Debias QA بشكل صحيح. ومع ذلك، نكتشف أن نماذج QG العصبية الأخيرة متحيزة نحو توليد الأسئلة ذات التداخل المعجمي العالي، والتي يمكن أن تضخيم تحيز DataSet. علاوة على ذلك، يكشف تحليلنا أن تكبير البيانات مع نماذج QG هذه تضعف بشكل متكرر الأداء على الأسئلة مع التداخل المعجمي المنخفض، مع تحسين ذلك على الأسئلة ذات التداخل المعجمي العالي. لمعالجة هذه المشكلة، نستخدم نهج مرادف يستند إلى استبدال أسئلة زيادة التداخل المعجمي المنخفض. نوضح أن نهج تكبير البيانات المقترح بسيط ولكنه فعال في التخفيف من مشكلة التدهور مع الأمثلة الاصطناعية 70K فقط.
Question answering (QA) models for reading comprehension have been demonstrated to exploit unintended dataset biases such as question--context lexical overlap. This hinders QA models from generalizing to under-represented samples such as questions with low lexical overlap. Question generation (QG), a method for augmenting QA datasets, can be a solution for such performance degradation if QG can properly debias QA datasets. However, we discover that recent neural QG models are biased towards generating questions with high lexical overlap, which can amplify the dataset bias. Moreover, our analysis reveals that data augmentation with these QG models frequently impairs the performance on questions with low lexical overlap, while improving that on questions with high lexical overlap. To address this problem, we use a synonym replacement-based approach to augment questions with low lexical overlap. We demonstrate that the proposed data augmentation approach is simple yet effective to mitigate the degradation problem with only 70k synthetic examples.
References used
https://aclanthology.org/
rate research
Read More
An overarching goal of natural language processing is to enable machines to communicate seamlessly with humans. However, natural language can be ambiguous or unclear. In cases of uncertainty, humans engage in an interactive process known as repair: a
sking questions and seeking clarification until their uncertainty is resolved. We propose a framework for building a visually grounded question-asking model capable of producing polar (yes-no) clarification questions to resolve misunderstandings in dialogue. Our model uses an expected information gain objective to derive informative questions from an off-the-shelf image captioner without requiring any supervised question-answer data. We demonstrate our model's ability to pose questions that improve communicative success in a goal-oriented 20 questions game with synthetic and human answerers.
NLP research in Hebrew has largely focused on morphology and syntax, where rich annotated datasets in the spirit of Universal Dependencies are available. Semantic datasets, however, are in short supply, hindering crucial advances in the development o
f NLP technology in Hebrew. In this work, we present ParaShoot, the first question answering dataset in modern Hebrew. The dataset follows the format and crowdsourcing methodology of SQuAD, and contains approximately 3000 annotated examples, similar to other question-answering datasets in low-resource languages. We provide the first baseline results using recently-released BERT-style models for Hebrew, showing that there is significant room for improvement on this task.
Although showing promising values to downstream applications, generating question and answer together is under-explored. In this paper, we introduce a novel task that targets question-answer pair generation from visual images. It requires not only ge
nerating diverse question-answer pairs but also keeping the consistency of them. We study different generation paradigms for this task and propose three models: the pipeline model, the joint model, and the sequential model. We integrate variational inference into these models to achieve diversity and consistency. We also propose region representation scaling and attention alignment to improve the consistency further. We finally devise an evaluator as a quantitative metric for consistency. We validate our approach on two benchmarks, VQA2.0 and Visual-7w, by automatically and manually evaluating diversity and consistency. Experimental results show the effectiveness of our models: they can generate diverse or consistent pairs. Moreover, this task can be used to improve visual question generation and visual question answering.
Question answering (QA) models use retriever and reader systems to answer questions. Reliance on training data by QA systems can amplify or reflect inequity through their responses. Many QA models, such as those for the SQuAD dataset, are trained and
tested on a subset of Wikipedia articles which encode their own biases and also reproduce real-world inequality. Understanding how training data affects bias in QA systems can inform methods to mitigate inequity. We develop two sets of questions for closed and open domain questions respectively, which use ambiguous questions to probe QA models for bias. We feed three deep-learning-based QA systems with our question sets and evaluate responses for bias via the metrics. Using our metrics, we find that open-domain QA models amplify biases more than their closed-domain counterparts and propose that biases in the retriever surface more readily due to greater freedom of choice.
Abstract Recent works have shown that language models (LM) capture different types of knowledge regarding facts or common sense. However, because no model is perfect, they still fail to provide appropriate answers in many cases. In this paper, we ask
the question, How can we know when language models know, with confidence, the answer to a particular query?'' We examine this question from the point of view of calibration, the property of a probabilistic model's predicted probabilities actually being well correlated with the probabilities of correctness. We examine three strong generative models---T5, BART, and GPT-2---and study whether their probabilities on QA tasks are well calibrated, finding the answer is a relatively emphatic no. We then examine methods to calibrate such models to make their confidence scores correlate better with the likelihood of correctness through fine-tuning, post-hoc probability modification, or adjustment of the predicted outputs or inputs. Experiments on a diverse range of datasets demonstrate the effectiveness of our methods. We also perform analysis to study the strengths and limitations of these methods, shedding light on further improvements that may be made in methods for calibrating LMs. We have released the code at https://github.com/jzbjyb/lm-calibration.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
