Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userTAPAS at SemEval-2021 Task 9: Reasoning over tables with intermediate pre-training
Tapas في Semeval-2021 Task 9: المنطق عبر الجداول مع التدريب المسبق المتوسط
514
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
نقدم مساهمة التاباس في المهمة المشتركة بشأن التحقق من البيان وإيجاد الأدلة مع الجداول (مهمة Semeval 2021 9، وانغ وآخرون (2021)). مهمة SEM Tab Factor Task A هي مهمة التصنيف بالاعتراف إذا تم إيصال بيان أو محايد أو دحض بمحتوى جدول معين. نعتمد نموذج تاباس ثنائي من Eisenschlos et al. (2020) لهذه المهمة. نحن نتعلم نماذج تصنيف ثنائية: نموذج أول للتنبؤ إذا كان عبارة محايدة أو غير محايدة وثانية واحدة للتنبؤ إذا كانت مستلمة أو دحض. نظرا لأن مجموعة التدريب المهمة المشتركة تحتوي فقط على أمثلة مستلمة أو دحض، فإننا نولد أمثلة محايدة اصطناعية لتدريب النموذج الأول. يتم تدريب كلا النموذجين مسبقا باستخدام بيانات Masklm موضوعية ومكافحة موظفة واصطناعية (Eisenschlos et al.، 2020) و Tabact (Chen et al.، 2020)، مجموعة بيانات استيابية طاولة كبيرة. نجد أن الأمثلة المحايدة الاصطناعية فعالة إلى حد ما في تدريب النموذج الأول، وتحقيق 68.03 اختبار F1 مقابل 60.47 من خط الأساس الأغلبية. في المرحلة الثانية، نجد أن التدريب المسبق على البيانات الوسيطة وتطويط التحمل يحسن النتائج عبر Masklm قبل التدريب (68.03 مقابل 57.01).
We present the TAPAS contribution to the Shared Task on Statement Verification and Evidence Finding with Tables (SemEval 2021 Task 9, Wang et al. (2021)). SEM TAB FACT Task A is a classification task of recognizing if a statement is entailed, neutral or refuted by the content of a given table. We adopt the binary TAPAS model of Eisenschlos et al. (2020) to this task. We learn two binary classification models: A first model to predict if a statement is neutral or non-neutral and a second one to predict if it is entailed or refuted. As the shared task training set contains only entailed or refuted examples, we generate artificial neutral examples to train the first model. Both models are pre-trained using a MASKLM objective, intermediate counter-factual and synthetic data (Eisenschlos et al., 2020) and TABFACT (Chen et al., 2020), a large table entailment dataset. We find that the artificial neutral examples are somewhat effective at training the first model, achieving 68.03 test F1 versus the 60.47 of a majority baseline. For the second stage, we find that the pre-training on the intermediate data and TABFACT improves the results over MASKLM pre-training (68.03 vs 57.01).
References used
https://aclanthology.org/
rate research
Read More
Tables are widely used in various kinds of documents to present information concisely. Understanding tables is a challenging problem that requires an understanding of language and table structure, along with numerical and logical reasoning. In this p
aper, we present our systems to solve Task 9 of SemEval-2021: Statement Verification and Evidence Finding with Tables (SEM-TAB-FACTS). The task consists of two subtasks: (A) Given a table and a statement, predicting whether the table supports the statement and (B) Predicting which cells in the table provide evidence for/against the statement. We fine-tune TAPAS (a model which extends BERT's architecture to capture tabular structure) for both the subtasks as it has shown state-of-the-art performance in various table understanding tasks. In subtask A, we evaluate how transfer learning and standardizing tables to have a single header row improves TAPAS' performance. In subtask B, we evaluate how different fine-tuning strategies can improve TAPAS' performance. Our systems achieve an F1 score of 67.34 in subtask A three-way classification, 72.89 in subtask A two-way classification, and 62.95 in subtask B.
This paper describes the system submitted in the SemEval-2021 Statement Verification and Evidence Finding with Tables task. The system relies on candidate generation for logical forms on the table based on keyword matching and dependency parsing on the claim statements.
Recently, there has been an interest in the research on factual verification and prediction over structured data like tables and graphs. To circumvent any false news incident, it is necessary to not only model and predict over structured data efficie
ntly but also to explain those predictions. In this paper, as the part of the SemEval-2021 Task 9, we tackle the problem of fact verification and evidence finding over tabular data. There are two subtasks, in which given a table and a statement/fact, the subtask A is to determine whether the statement is inferred from the tabular data and the subtask B is to determine which cells in the table provide evidence for the former subtask. We make a comparison of the baselines and state of the art approaches over the given SemTabFact dataset. We also propose a novel approach CellBERT to solve the task of evidence finding, as a form of Natural Language Inference task. We obtain a 3-way F1 score of 0.69 on subtask A and an F1 score of 0.65 on subtask B.
This paper describes our approach for Task 9 of SemEval 2021: Statement Verification and Evidence Finding with Tables. We participated in both subtasks, namely statement verification and evidence finding. For the subtask of statement verification, we
extend the TAPAS model to adapt to the unknown' class of statements by finetuning it on an augmented version of the task data. For the subtask of evidence finding, we finetune the DistilBERT model in a Siamese setting.
Current NLP models are predominantly trained through a two-stage pre-train then fine-tune'' pipeline. Prior work has shown that inserting an intermediate pre-training stage, using heuristic masking policies for masked language modeling (MLM), can sig
nificantly improve final performance. However, it is still unclear (1) in what cases such intermediate pre-training is helpful, (2) whether hand-crafted heuristic objectives are optimal for a given task, and (3) whether a masking policy designed for one task is generalizable beyond that task. In this paper, we perform a large-scale empirical study to investigate the effect of various masking policies in intermediate pre-training with nine selected tasks across three categories. Crucially, we introduce methods to automate the discovery of optimal masking policies via direct supervision or meta-learning. We conclude that the success of intermediate pre-training is dependent on appropriate pre-train corpus, selection of output format (i.e., masked spans or full sentence), and clear understanding of the role that MLM plays for the downstream task. In addition, we find our learned masking policies outperform the heuristic of masking named entities on TriviaQA, and policies learned from one task can positively transfer to other tasks in certain cases, inviting future research in this direction.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
