Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userMitigating False-Negative Contexts in Multi-document Question Answering with Retrieval Marginalization
التخفيف من السياقات الخاطئة السلبية في سؤال متعدد الوثائق الرد مع تهميش الاسترجاع
152
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
غالبا ما تعتمد مهام الإجابة على الأسئلة التي تتطلب معلومات من وثائق متعددة على نموذج استرجاع لتحديد المعلومات ذات الصلة للتفكير. يتم تدريب نموذج الاسترجاع عادة على تعظيم احتمالية الأدلة الداعمة المسمى. ومع ذلك، عند الاسترجاع من نصائح نصية كبيرة مثل Wikipedia، غالبا ما يتم الحصول على الإجابة الصحيحة من مرشحين أدلة متعددة. علاوة على ذلك، لا يتم تصنيف جميع هؤلاء المرشحين على أنها إيجابية أثناء التوضيحية، مما يجعل إشارة التدريب ضعيفة وصاخبة. يتم تفاقم هذه المشكلة عندما تكون الأسئلة غير مرضية أو عندما تكون الإجابات منطقية، نظرا لأن النموذج لا يمكن أن يعتمد على التداخل المعجمي لإجراء اتصال بين الإجابة والدليل الدعم. نقوم بتطوير مخففة جديدة لاسترجاع المحدد الذي يتعامل مع الاستعلامات غير المجهولية، ونظهر أن التهميش حول هذه المجموعة أثناء التدريب يسمح للنموذج لتخفيف السلبيات الخاطئة في دعم التعليقات التوضيحية للأدلة. نحن نختبر طريقتنا في مجموعات بيانات QA متعددة الوثائق، IIRC و hotpotqa. على IIRC، نظهر أن النمذجة المشتركة مع التهميش يحسن أداء نموذجي بنسبة 5.5 F1 ونحقق أداء جديد من بين الفنون 50.5 F1. نوضح أيضا أن تهميش استرجاع ينتج عنه تحسن 4.1 ضمير QA F1 على خط الأساس غير المهمش في HOTPOTQA في إعداد FullWiki.
Question Answering (QA) tasks requiring information from multiple documents often rely on a retrieval model to identify relevant information for reasoning. The retrieval model is typically trained to maximize the likelihood of the labeled supporting evidence. However, when retrieving from large text corpora such as Wikipedia, the correct answer can often be obtained from multiple evidence candidates. Moreover, not all such candidates are labeled as positive during annotation, rendering the training signal weak and noisy. This problem is exacerbated when the questions are unanswerable or when the answers are Boolean, since the model cannot rely on lexical overlap to make a connection between the answer and supporting evidence. We develop a new parameterization of set-valued retrieval that handles unanswerable queries, and we show that marginalizing over this set during training allows a model to mitigate false negatives in supporting evidence annotations. We test our method on two multi-document QA datasets, IIRC and HotpotQA. On IIRC, we show that joint modeling with marginalization improves model performance by 5.5 F1 points and achieves a new state-of-the-art performance of 50.5 F1. We also show that retrieval marginalization results in 4.1 QA F1 improvement over a non-marginalized baseline on HotpotQA in the fullwiki setting.
References used
https://aclanthology.org/
rate research
Read More
We tackle multi-choice question answering. Acquiring related commonsense knowledge to the question and options facilitates the recognition of the correct answer. However, the current reasoning models suffer from the noises in the retrieved knowledge.
In this paper, we propose a novel encoding method which is able to conduct interception and soft filtering. This contributes to the harvesting and absorption of representative information with less interference from noises. We experiment on CommonsenseQA. Experimental results illustrate that our method yields substantial and consistent improvements compared to the strong Bert, RoBERTa and Albert-based baselines.
Recent advances in open-domain QA have led to strong models based on dense retrieval, but only focused on retrieving textual passages. In this work, we tackle open-domain QA over tables for the first time, and show that retrieval can be improved by a
retriever designed to handle tabular context. We present an effective pre-training procedure for our retriever and improve retrieval quality with mined hard negatives. As relevant datasets are missing, we extract a subset of Natural Questions (Kwiatkowski et al., 2019) into a Table QA dataset. We find that our retriever improves retrieval results from 72.0 to 81.1 recall@10 and end-to-end QA results from 33.8 to 37.7 exact match, over a BERT based retriever.
Question answering (QA) models use retriever and reader systems to answer questions. Reliance on training data by QA systems can amplify or reflect inequity through their responses. Many QA models, such as those for the SQuAD dataset, are trained and
tested on a subset of Wikipedia articles which encode their own biases and also reproduce real-world inequality. Understanding how training data affects bias in QA systems can inform methods to mitigate inequity. We develop two sets of questions for closed and open domain questions respectively, which use ambiguous questions to probe QA models for bias. We feed three deep-learning-based QA systems with our question sets and evaluate responses for bias via the metrics. Using our metrics, we find that open-domain QA models amplify biases more than their closed-domain counterparts and propose that biases in the retriever surface more readily due to greater freedom of choice.
Many datasets have been created for training reading comprehension models, and a natural question is whether we can combine them to build models that (1) perform better on all of the training datasets and (2) generalize and transfer better to new dat
asets. Prior work has addressed this goal by training one network simultaneously on multiple datasets, which works well on average but is prone to over- or under-fitting different sub- distributions and might transfer worse compared to source models with more overlap with the target dataset. Our approach is to model multi-dataset question answering with an ensemble of single-dataset experts, by training a collection of lightweight, dataset-specific adapter modules (Houlsby et al., 2019) that share an underlying Transformer model. We find that these Multi-Adapter Dataset Experts (MADE) outperform all our baselines in terms of in-distribution accuracy, and simple methods based on parameter-averaging lead to better zero-shot generalization and few-shot transfer performance, offering a strong and versatile starting point for building new reading comprehension systems.
In simple open-domain question answering (QA), dense retrieval has become one of the standard approaches for retrieving the relevant passages to infer an answer. Recently, dense retrieval also achieved state-of-the-art results in multi-hop QA, where
aggregating information from multiple pieces of information and reasoning over them is required. Despite their success, dense retrieval methods are computationally intensive, requiring multiple GPUs to train. In this work, we introduce a hybrid (lexical and dense) retrieval approach that is highly competitive with the state-of-the-art dense retrieval models, while requiring substantially less computational resources. Additionally, we provide an in-depth evaluation of dense retrieval methods on limited computational resource settings, something that is missing from the current literature.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
