Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userSemantic Answer Similarity for Evaluating Question Answering Models
الإجابة الدلالية التشابه لتقييم نماذج الإجابة على الإجابة
332
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
يقارن تقييم نماذج الرد على الأسئلة التوضيحية حول التوقعات النموذجية. ومع ذلك، اعتبارا من اليوم، فإن هذه المقارنة تعتمد في الغالب معجمية، وبالتالي تفتقد الإجابات التي لا تحتوي على تداخل جذري ولكن لا تزال مماثلة متشابهة دلالة، وبالتالي علاج الإجابات الصحيحة على أنها خاطئة. يعيق هذا التقليل من الأداء الحقيقي للنماذج قبول المستخدم في التطبيقات ويعقد مقارنة عادلة من النماذج المختلفة. لذلك، هناك حاجة إلى متري تقييم يعتمد على دلالات بدلا من تشابه السلسلة الخالصة. في هذه الورقة القصيرة، نقدم SAS، وهي متري مقرها في التشفير لتقدير تشابه الإجابة الدلالية، ومقارنتها بسبعة مقاييس موجودة. تحقيقا لهذه الغاية، نقوم بإنشاء مجموعة بيانات تقييم اللغة الإنجليزية ثلاثية وألمانية تحتوي على أزواج من الإجابات جنبا إلى جنب مع الحكم البشري من التشابه الدلالي، والتي نصرح لها جنبا إلى جنب مع تنفيذ Metric SAS والتجارب. نجد أن مقاييس التشابه الدلالي القائم على نماذج المحولات الأخيرة ترتبط بشكل أفضل بكثير مع الحكم البشري من مقاييس التشابه المعجمية التقليدية على مجموعات بياناتنا التي أنشأت حديثا ومجموعة بيانات واحدة من العمل ذي الصلة.
The evaluation of question answering models compares ground-truth annotations with model predictions. However, as of today, this comparison is mostly lexical-based and therefore misses out on answers that have no lexical overlap but are still semantically similar, thus treating correct answers as false. This underestimation of the true performance of models hinders user acceptance in applications and complicates a fair comparison of different models. Therefore, there is a need for an evaluation metric that is based on semantics instead of pure string similarity. In this short paper, we present SAS, a cross-encoder-based metric for the estimation of semantic answer similarity, and compare it to seven existing metrics. To this end, we create an English and a German three-way annotated evaluation dataset containing pairs of answers along with human judgment of their semantic similarity, which we release along with an implementation of the SAS metric and the experiments. We find that semantic similarity metrics based on recent transformer models correlate much better with human judgment than traditional lexical similarity metrics on our two newly created datasets and one dataset from related work.
References used
https://aclanthology.org/
rate research
Read More
In education, quiz questions have become an important tool for assessing the knowledge of students. Yet, manually preparing such questions is a tedious task, and thus automatic question generation has been proposed as a possible alternative. So far,
the vast majority of research has focused on generating the question text, relying on question answering datasets with readily picked answers, and the problem of how to come up with answer candidates in the first place has been largely ignored. Here, we aim to bridge this gap. In particular, we propose a model that can generate a specified number of answer candidates for a given passage of text, which can then be used by instructors to write questions manually or can be passed as an input to automatic answer-aware question generators. Our experiments show that our proposed answer candidate generation model outperforms several baselines.
Abstract A desirable property of a reference-based evaluation metric that measures the content quality of a summary is that it should estimate how much information that summary has in common with a reference. Traditional text overlap based metrics su
ch as ROUGE fail to achieve this because they are limited to matching tokens, either lexically or via embeddings. In this work, we propose a metric to evaluate the content quality of a summary using question-answering (QA). QA-based methods directly measure a summary's information overlap with a reference, making them fundamentally different than text overlap metrics. We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval. QAEval outperforms current state-of-the-art metrics on most evaluations using benchmark datasets, while being competitive on others due to limitations of state-of-the-art models. Through a careful analysis of each component of QAEval, we identify its performance bottlenecks and estimate that its potential upper-bound performance surpasses all other automatic metrics, approaching that of the gold-standard Pyramid Method.1
We present an information retrieval-based question answer system to answer legal questions. The system is not limited to a predefined set of questions or patterns and uses both sparse vector search and embeddings for input to a BERT-based answer re-r
anking system. A combination of general domain and legal domain data is used for training. This natural question answering system is in production and is used commercially.
Weakly-supervised table question-answering (TableQA) models have achieved state-of-art performance by using pre-trained BERT transformer to jointly encoding a question and a table to produce structured query for the question. However, in practical se
ttings TableQA systems are deployed over table corpora having topic and word distributions quite distinct from BERT's pretraining corpus. In this work we simulate the practical topic shift scenario by designing novel challenge benchmarks WikiSQL-TS and WikiTable-TS, consisting of train-dev-test splits in five distinct topic groups, based on the popular WikiSQL and WikiTable-Questions datasets. We empirically show that, despite pre-training on large open-domain text, performance of models degrades significantly when they are evaluated on unseen topics. In response, we propose T3QA (Topic Transferable Table Question Answering) a pragmatic adaptation framework for TableQA comprising of: (1) topic-specific vocabulary injection into BERT, (2) a novel text-to-text transformer generator (such as T5, GPT2) based natural language question generation pipeline focused on generating topic-specific training data, and (3) a logical form re-ranker. We show that T3QA provides a reasonably good baseline for our topic shift benchmarks. We believe our topic split benchmarks will lead to robust TableQA solutions that are better suited for practical deployment
NLP research in Hebrew has largely focused on morphology and syntax, where rich annotated datasets in the spirit of Universal Dependencies are available. Semantic datasets, however, are in short supply, hindering crucial advances in the development o
f NLP technology in Hebrew. In this work, we present ParaShoot, the first question answering dataset in modern Hebrew. The dataset follows the format and crowdsourcing methodology of SQuAD, and contains approximately 3000 annotated examples, similar to other question-answering datasets in low-resource languages. We provide the first baseline results using recently-released BERT-style models for Hebrew, showing that there is significant room for improvement on this task.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
