Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userWinoLogic: A Zero-Shot Logic-based Diagnostic Dataset for Winograd Schema Challenge
Winologic: مجموعة بيانات التشخيص المستندة إلى منطق صفر لقطة
310
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
دعا النجاح الأخير لنماذج اللغة العصبية (NLMS) على تحدي مخطط Winograd إلى مزيد من التحقيق في قدرة المنطق المنطقي لهذه النماذج. تعتمد مجموعات البيانات التشخيصية السابقة على مصادر الحشد التي تفشل في توفير أمر مناسب متماسك لحل مشاكل WSC. لتحسين تقييم NLMS بشكل أفضل، نقترح إطارا يستند إلى المنطق يركز على معرفة المناخية عالية الجودة. على وجه التحديد، نقوم بتحديد وجمع صيغ المعرفة الرسمية التي تم التحقق منها بواسطة Theorem Brovers وترجمة هذه الصيغ إلى جمل لغة طبيعية. بناء على جمل المعرفة الحقيقية هذه، يتم إنشاء تلك الخوذة الخاطئة. نقترح مجموعة بيانات جديدة تسمى Winologic مع هذه الجمل. نظرا لمشكلة Winologic، تحتاج NLMS إلى تحديد ما إذا كانت جمل المعرفة المعقولة يمكن أن تحل مشاكل WSC المقابلة بشكل صحيح في إعداد طلقة صفرية. نطلب أيضا عن النذوي البشري للتحقق من صحة نولولوجية لضمان أن يكون مقبول من الإنسان. تشير التجارب إلى أن NLMS ما زالت تكافح لفهم معرفة المنظمات كإنسان، مما يشير إلى المبالغة في تقدير قدرتهم في التفكير.
The recent success of neural language models (NLMs) on the Winograd Schema Challenge has called for further investigation of the commonsense reasoning ability of these models. Previous diagnostic datasets rely on crowd-sourcing which fails to provide coherent commonsense crucial for solving WSC problems. To better evaluate NLMs, we propose a logic-based framework that focuses on high-quality commonsense knowledge. Specifically, we identify and collect formal knowledge formulas verified by theorem provers and translate such formulas into natural language sentences. Based on these true knowledge sentences, adversarial false ones are generated. We propose a new dataset named WinoLogic with these sentences. Given a problem in WinoLogic, NLMs need to decide whether the plausible knowledge sentences could correctly solve the corresponding WSC problems in a zero-shot setting. We also ask human annotators to validate WinoLogic to ensure it is human-agreeable. Experiments show that NLMs still struggle to comprehend commonsense knowledge as humans do, indicating that their reasoning ability could have been overestimated.
References used
https://aclanthology.org/
rate research
Read More
Web search is an essential way for humans to obtain information, but it's still a great challenge for machines to understand the contents of web pages. In this paper, we introduce the task of web-based structural reading comprehension. Given a web pa
ge and a question about it, the task is to find an answer from the web page. This task requires a system not only to understand the semantics of texts but also the structure of the web page. Moreover, we proposed WebSRC, a novel Web-based Structural Reading Comprehension dataset. WebSRC consists of 400K question-answer pairs, which are collected from 6.4K web pages with corresponding HTML source code, screenshots, and metadata. Each question in WebSRC requires a certain structural understanding of a web page to answer, and the answer is either a text span on the web page or yes/no. We evaluate various strong baselines on our dataset to show the difficulty of our task. We also investigate the usefulness of structural information and visual features. Our dataset and baselines have been publicly available.
In clinical studies, chatbots mimicking doctor-patient interactions are used for collecting information about the patient's health state. Later, this information needs to be processed and structured for the doctor. One way to organize it is by automa
tically filling the questionnaires from the human-bot conversation. It would help the doctor to spot the possible issues. Since there is no such dataset available for this task and its collection is costly and sensitive, we explore the capacities of state-of-the-art zero-shot models for question answering, textual inference, and text classification. We provide a detailed analysis of the results and propose further directions for clinical questionnaire filling.
Stance detection on social media can help to identify and understand slanted news or commentary in everyday life. In this work, we propose a new model for zero-shot stance detection on Twitter that uses adversarial learning to generalize across topic
s. Our model achieves state-of-the-art performance on a number of unseen test topics with minimal computational costs. In addition, we extend zero-shot stance detection to topics not previously considered, highlighting future directions for zero-shot transfer.
We introduce MULTI-EURLEX, a new multilingual dataset for topic classification of legal documents. The dataset comprises 65k European Union (EU) laws, officially translated in 23 languages, annotated with multiple labels from the EUROVOC taxonomy. We
highlight the effect of temporal concept drift and the importance of chronological, instead of random splits. We use the dataset as a testbed for zero-shot cross-lingual transfer, where we exploit annotated training documents in one language (source) to classify documents in another language (target). We find that fine-tuning a multilingually pretrained model (XLM-ROBERTA, MT5) in a single source language leads to catastrophic forgetting of multilingual knowledge and, consequently, poor zero-shot transfer to other languages. Adaptation strategies, namely partial fine-tuning, adapters, BITFIT, LNFIT, originally proposed to accelerate fine-tuning for new end-tasks, help retain multilingual knowledge from pretraining, substantially improving zero-shot cross-lingual transfer, but their impact also depends on the pretrained model used and the size of the label set.
We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-base
d architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
