Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userOn the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
على التحيز الاستقرائي للنمذجة اللغوية الملثمان: من الإحصاء إلى التبعيات النحوية
374
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
ندرس كيف يمكن أن يؤدي اخفاء وتنبؤ الرموز في الأزياء غير المنشأة إلى الهياكل اللغوية ومكاسب أداء المصب. اقترحت النظريات الحديثة أن نماذج اللغة المحددة مسبقا تكتسب تحيزات حثي مفيدة من خلال الأقنعة التي تعمل ضمنيا كتخفيض كتفل لمهام المصب. في حين أننا ننظر إلى أن نجاح استراتيجية الاخفاء العشوائية المستخدمة في الممارسة لا يمكن تفسيره بواسطة أقنعة مثل كلوزي وحدها. نحن نبني أقنعة تشبه الكتبة باستخدام المعجمات الخاصة بمهام المهام لثلاث مجموعات بيانات تصنيف مختلفة وإظهار أن غالبية مكاسب الأداء المسبدة مسبقا تأتي من أقنعة عامة لا ترتبط مع المعجم. لشرح النجاح التجريبي لهذه الأقنعة العامة هذه، نوضح مراسلات بين هدف اللغة المعقدة (MLM) والأساليب الموجودة لتعلم التبعيات الإحصائية في النماذج الرسومية. باستخدام هذا، نستمد طريقة لاستخراج هذه التبعيات الإحصائية المستفادة في MLMS وإظهار أن هذه التبعيات ترميز تحيزات حثي مفيدة في شكل هياكل النحوية. في تقييم تحليل غير مدفوع، ببساطة تشكيل شجرة ممتدة كحد أدنى على هيكل الاعتماد الإحصائي الضمني تتفوق على طريقة كلاسيكية لتحليل غير معروض (58.74 مقابل 55.91 Uuas).
We study how masking and predicting tokens in an unsupervised fashion can give rise to linguistic structures and downstream performance gains. Recent theories have suggested that pretrained language models acquire useful inductive biases through masks that implicitly act as cloze reductions for downstream tasks. While appealing, we show that the success of the random masking strategy used in practice cannot be explained by such cloze-like masks alone. We construct cloze-like masks using task-specific lexicons for three different classification datasets and show that the majority of pretrained performance gains come from generic masks that are not associated with the lexicon. To explain the empirical success of these generic masks, we demonstrate a correspondence between the Masked Language Model (MLM) objective and existing methods for learning statistical dependencies in graphical models. Using this, we derive a method for extracting these learned statistical dependencies in MLMs and show that these dependencies encode useful inductive biases in the form of syntactic structures. In an unsupervised parsing evaluation, simply forming a minimum spanning tree on the implied statistical dependence structure outperforms a classic method for unsupervised parsing (58.74 vs. 55.91 UUAS).
References used
https://aclanthology.org/
rate research
Read More
The lack of publicly available evaluation data for low-resource languages limits progress in Spoken Language Understanding (SLU). As key tasks like intent classification and slot filling require abundant training data, it is desirable to reuse existi
ng data in high-resource languages to develop models for low-resource scenarios. We introduce xSID, a new benchmark for cross-lingual (x) Slot and Intent Detection in 13 languages from 6 language families, including a very low-resource dialect. To tackle the challenge, we propose a joint learning approach, with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer. We study two setups which differ by type and language coverage of the pre-trained embeddings. Our results show that jointly learning the main tasks with masked language modeling is effective for slots, while machine translation transfer works best for intent classification.
We describe the second IWPT task on end-to-end parsing from raw text to Enhanced Universal Dependencies. We provide details about the evaluation metrics and the datasets used for training and evaluation. We compare the approaches taken by participating teams and discuss the results of the shared task, also in comparison with the first edition of this task.
Fully understanding narratives often requires identifying events in the context of whole documents and modeling the event relations. However, document-level event extraction is a challenging task as it requires the extraction of event and entity core
ference, and capturing arguments that span across different sentences. Existing works on event extraction usually confine on extracting events from single sentences, which fail to capture the relationships between the event mentions at the scale of a document, as well as the event arguments that appear in a different sentence than the event trigger. In this paper, we propose an end-to-end model leveraging Deep Value Networks (DVN), a structured prediction algorithm, to efficiently capture cross-event dependencies for document-level event extraction. Experimental results show that our approach achieves comparable performance to CRF-based models on ACE05, while enjoys significantly higher computational efficiency.
Fine-tuned language models have been shown to exhibit biases against protected groups in a host of modeling tasks such as text classification and coreference resolution. Previous works focus on detecting these biases, reducing bias in data representa
tions, and using auxiliary training objectives to mitigate bias during fine-tuning. Although these techniques achieve bias reduction for the task and domain at hand, the effects of bias mitigation may not directly transfer to new tasks, requiring additional data collection and customized annotation of sensitive attributes, and re-evaluation of appropriate fairness metrics. We explore the feasibility and benefits of upstream bias mitigation (UBM) for reducing bias on downstream tasks, by first applying bias mitigation to an upstream model through fine-tuning and subsequently using it for downstream fine-tuning. We find, in extensive experiments across hate speech detection, toxicity detection and coreference resolution tasks over various bias factors, that the effects of UBM are indeed transferable to new downstream tasks or domains via fine-tuning, creating less biased downstream models than directly fine-tuning on the downstream task or transferring from a vanilla upstream model. Though challenges remain, we show that UBM promises more efficient and accessible bias mitigation in LM fine-tuning.
In this paper, we study ethnic bias and how it varies across languages by analyzing and mitigating ethnic bias in monolingual BERT for English, German, Spanish, Korean, Turkish, and Chinese. To observe and quantify ethnic bias, we develop a novel met
ric called Categorical Bias score. Then we propose two methods for mitigation; first using a multilingual model, and second using contextual word alignment of two monolingual models. We compare our proposed methods with monolingual BERT and show that these methods effectively alleviate the ethnic bias. Which of the two methods works better depends on the amount of NLP resources available for that language. We additionally experiment with Arabic and Greek to verify that our proposed methods work for a wider variety of languages.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
