Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userOn the Transferability of Minimal Prediction Preserving Inputs in Question Answering
على تحويل الحد الأدنى من المدخلات الاحتفاظ بالتنبؤ في الإجابة
232
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
يعمل العمل الأخير (فنغ وآخرون.، 2018) وجود شظايا مدخلات قصيرة غير قابلة للتفسير التي تحقق ثقة عالية ودقة في النماذج العصبية. نشير إلى هذه المدخلات الحد الأدنى للحفاظ على التنبؤ (MPPIS). في سياق الإجابة على السؤال، نحن نحقق في الفرضيات المتنافسة لوجود MPPIs، بما في ذلك ضعف المعايرة الخلفية للنماذج العصبية، ونقص الاحيلات، وتحيز DataSet "(حيث يتعلم نموذج الحضور إلى العظة الزائفة غير الجماعية بيانات التدريب). نكتشف التحير الثابت في MPPIS إلى بذور التدريب العشوائي، والهندسة المعمارية النموذجية، ومجال التدريب، والمجال التدريبي. إظهار MPPIS إمكانية نقل رائعة عبر المجالات التي تحقق أداء أعلى بكثير من استفسارات قصيرة مماثلة. بالإضافة إلى ذلك، فشل معاقبة الثقة الزائدة في MPPIS تحسين إما التعميم أو المتانة الخصومة. تشير هذه النتائج إلى إمكانية تفسير MPPIS غير كافية لتوصيف قدرة التعميم لهذه النماذج. نأمل أن يشجع هذا التحقيق المركز تحليلا منهجيا للسلوك النموذجي خارج التوزيع القابل للتفسير البشري للأمثلة.
Recent work (Feng et al., 2018) establishes the presence of short, uninterpretable input fragments that yield high confidence and accuracy in neural models. We refer to these as Minimal Prediction Preserving Inputs (MPPIs). In the context of question answering, we investigate competing hypotheses for the existence of MPPIs, including poor posterior calibration of neural models, lack of pretraining, and dataset bias'' (where a model learns to attend to spurious, non-generalizable cues in the training data). We discover a perplexing invariance of MPPIs to random training seed, model architecture, pretraining, and training domain. MPPIs demonstrate remarkable transferability across domains achieving significantly higher performance than comparably short queries. Additionally, penalizing over-confidence on MPPIs fails to improve either generalization or adversarial robustness. These results suggest the interpretability of MPPIs is insufficient to characterize generalization capacity of these models. We hope this focused investigation encourages more systematic analysis of model behavior outside of the human interpretable distribution of examples.
References used
https://aclanthology.org/
rate research
Read More
In open-domain question answering (QA), retrieve-and-read mechanism has the inherent benefit of interpretability and the easiness of adding, removing, or editing knowledge compared to the parametric approaches of closed-book QA models. However, it is
also known to suffer from its large storage footprint due to its document corpus and index. Here, we discuss several orthogonal strategies to drastically reduce the footprint of a retrieve-and-read open-domain QA system by up to 160x. Our results indicate that retrieve-and-read can be a viable option even in a highly constrained serving environment such as edge devices, as we show that it can achieve better accuracy than a purely parametric model with comparable docker-level system size.
The evaluation of question answering models compares ground-truth annotations with model predictions. However, as of today, this comparison is mostly lexical-based and therefore misses out on answers that have no lexical overlap but are still semanti
cally similar, thus treating correct answers as false. This underestimation of the true performance of models hinders user acceptance in applications and complicates a fair comparison of different models. Therefore, there is a need for an evaluation metric that is based on semantics instead of pure string similarity. In this short paper, we present SAS, a cross-encoder-based metric for the estimation of semantic answer similarity, and compare it to seven existing metrics. To this end, we create an English and a German three-way annotated evaluation dataset containing pairs of answers along with human judgment of their semantic similarity, which we release along with an implementation of the SAS metric and the experiments. We find that semantic similarity metrics based on recent transformer models correlate much better with human judgment than traditional lexical similarity metrics on our two newly created datasets and one dataset from related work.
Morphological rules with various levels of specificity can be learned from example lexemes by recursive application of minimal generalization (Albright and Hayes, 2002, 2003).A model that learns rules solely through minimal generalization was used to
predict average human wug-test ratings from German, English, and Dutch in the SIGMORPHON-UniMorph 2021 Shared Task, with competitive results. Some formal properties of the minimal generalization operation were proved. An automatic method was developed to create wug-test stimuli for future experiments that investigate whether the model's morphological generalizations are too minimal.
NLP research in Hebrew has largely focused on morphology and syntax, where rich annotated datasets in the spirit of Universal Dependencies are available. Semantic datasets, however, are in short supply, hindering crucial advances in the development o
f NLP technology in Hebrew. In this work, we present ParaShoot, the first question answering dataset in modern Hebrew. The dataset follows the format and crowdsourcing methodology of SQuAD, and contains approximately 3000 annotated examples, similar to other question-answering datasets in low-resource languages. We provide the first baseline results using recently-released BERT-style models for Hebrew, showing that there is significant room for improvement on this task.
We present an information retrieval-based question answer system to answer legal questions. The system is not limited to a predefined set of questions or patterns and uses both sparse vector search and embeddings for input to a BERT-based answer re-r
anking system. A combination of general domain and legal domain data is used for training. This natural question answering system is in production and is used commercially.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
