Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userBehavior of Modern Pre-trained Language Models Using the Example of Probing Tasks
سلوك نماذج اللغة المدربة مسبقا مسبقا باستخدام مثال المهام التحقيق
363
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تحدث نماذج اللغات القائمة على المحولات الحديثة ثورة في NLP. ومع ذلك، كانت الدراسات الحالية في النمذجة اللغوية مع بيرت تقتصر في الغالب على المواد باللغة الإنجليزية ولا تدفع اهتماما كافيا لمعرفة اللغة الضمنية باللغة، مثل الأدوار الدلالية والتفترض واللبن، والتي يمكن الحصول عليها من قبل النموذج أثناء التدريب. وبالتالي، فإن الهدف من هذه الدراسة هو فحص السلوك لنموذج الموديل في مهمة النمذجة اللغوية الملثمين ولتقديم التفسير اللغوي إلى الآثار والأخطاء غير المتوقعة التي ينتجها النموذج. لهذا الغرض، استخدمنا مجموعة بيانات جديدة باللغة الروسية بناء على النصوص التعليمية للمتعلمين باللغة الروسية والمصفحة بمساعدة الشقوق الوطنية للغة الروسية. من حيث مقاييس الجودة (نسبة الكلمات، ذات الصلة دلالة الكلمة المستهدفة)، يتم التعرف على بيرت متعددة اللغات كأفضل نموذج. بشكل عام، كل طراز لديه نقاط قوة متميزة فيما يتعلق بظاهرة لغوية معينة. هذه الملاحظات لها آثار ذات مغزى على البحث في اللغويات المطبقة والبيتاجوجية، والمساهمة في تطوير نظام الحوار، وجعل التمارين التلقائية، وتجول النص، ويمكن أن يحتمل أن يحسن جودة التقنيات اللغوية الحالية
Modern transformer-based language models are revolutionizing NLP. However, existing studies into language modelling with BERT have been mostly limited to English-language material and do not pay enough attention to the implicit knowledge of language, such as semantic roles, presupposition and negations, that can be acquired by the model during training. Thus, the aim of this study is to examine behavior of the model BERT in the task of masked language modelling and to provide linguistic interpretation to the unexpected effects and errors produced by the model. For this purpose, we used a new Russian-language dataset based on educational texts for learners of Russian and annotated with the help of the National Corpus of the Russian language. In terms of quality metrics (the proportion of words, semantically related to the target word), the multilingual BERT is recognized as the best model. Generally, each model has distinct strengths in relation to a certain linguistic phenomenon. These observations have meaningful implications for research into applied linguistics and pedagogy, contribute to dialogue system development, automatic exercise making, text generation and potentially could improve the quality of existing linguistic technologies
References used
https://aclanthology.org/
rate research
Read More
Pretrained language models (PTLMs) yield state-of-the-art performance on many natural language processing tasks, including syntax, semantics and commonsense. In this paper, we focus on identifying to what extent do PTLMs capture semantic attributes a
nd their values, e.g., the correlation between rich and high net worth. We use PTLMs to predict masked tokens using patterns and lists of items from Wikidata in order to verify how likely PTLMs encode semantic attributes along with their values. Such inferences based on semantics are intuitive for humans as part of our language understanding. Since PTLMs are trained on large amount of Wikipedia data we would assume that they can generate similar predictions, yet our findings reveal that PTLMs are still much worse than humans on this task. We show evidence and analysis explaining how to exploit our methodology to integrate better context and semantics into PTLMs using knowledge bases.
Pre-trained language models (PrLM) have to carefully manage input units when training on a very large text with a vocabulary consisting of millions of words. Previous works have shown that incorporating span-level information over consecutive words i
n pre-training could further improve the performance of PrLMs. However, given that span-level clues are introduced and fixed in pre-training, previous methods are time-consuming and lack of flexibility. To alleviate the inconvenience, this paper presents a novel span fine-tuning method for PrLMs, which facilitates the span setting to be adaptively determined by specific downstream tasks during the fine-tuning phase. In detail, any sentences processed by the PrLM will be segmented into multiple spans according to a pre-sampled dictionary. Then the segmentation information will be sent through a hierarchical CNN module together with the representation outputs of the PrLM and ultimately generate a span-enhanced representation. Experiments on GLUE benchmark show that the proposed span fine-tuning method significantly enhances the PrLM, and at the same time, offer more flexibility in an efficient way.
Attention-based pre-trained language models such as GPT-2 brought considerable progress to end-to-end dialogue modelling. However, they also present considerable risks for task-oriented dialogue, such as lack of knowledge grounding or diversity. To a
ddress these issues, we introduce modified training objectives for language model finetuning, and we employ massive data augmentation via back-translation to increase the diversity of the training data. We further examine the possibilities of combining data from multiples sources to improve performance on the target dataset. We carefully evaluate our contributions with both human and automatic methods. Our model substantially outperforms the baseline on the MultiWOZ data and shows competitive performance with state of the art in both automatic and human evaluation.
Existing work on probing of pretrained language models (LMs) has predominantly focused on sentence-level syntactic tasks. In this paper, we introduce document-level discourse probing to evaluate the ability of pretrained LMs to capture document-level
relations. We experiment with 7 pretrained LMs, 4 languages, and 7 discourse probing tasks, and find BART to be overall the best model at capturing discourse --- but only in its encoder, with BERT performing surprisingly well as the baseline model. Across the different models, there are substantial differences in which layers best capture discourse information, and large disparities between models.
Legal texts routinely use concepts that are difficult to understand. Lawyers elaborate on the meaning of such concepts by, among other things, carefully investigating how they have been used in the past. Finding text snippets that mention a particula
r concept in a useful way is tedious, time-consuming, and hence expensive. We assembled a data set of 26,959 sentences, coming from legal case decisions, and labeled them in terms of their usefulness for explaining selected legal concepts. Using the dataset we study the effectiveness of transformer models pre-trained on large language corpora to detect which of the sentences are useful. In light of models' predictions, we analyze various linguistic properties of the explanatory sentences as well as their relationship to the legal concept that needs to be explained. We show that the transformer-based models are capable of learning surprisingly sophisticated features and outperform the prior approaches to the task.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
