Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userShaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations
تهتز الأشجار النحوية في شارع السمسم: التحقيق متعدد اللغات مع اضطرابات يمكن السيطرة عليها
148
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
اعتمدت الأبحاث الحديثة حقل تجريبي جديد يترکن حول مفهوم اضطرابات النصوص التي كشفت عن ترتيب الكلمات الخلفي ليس لها تأثير كبير على أداء نماذج اللغة القائمة على المحولات في العديد من مهام NLP. تتناقض هذه النتائج بالفهم المشترك لكيفية تشفير النماذج من المعلومات الهرمية والهيكلية وحتى السؤال إذا تم تصميم أمر Word مع Adgeddings الموضع. تحقيقا لهذه الغاية، تقترح هذه الورقة تسع مجموعات بيانات للتحقيق التي تنظمها نوع اضطراب النص الذي يمكن السيطرة عليه لثلاثة لغات داخلية من الهند مع درجة متفاوتة من مرونة ترتيب الكلمات: الإنجليزية والسويدية والروسية. استنادا إلى تحليل التحقيق لنماذج M-Bert و M-Bart، نبلغ أن الحساسية النحوية تعتمد على أهداف اللغة والنموذج قبل التدريب. نجد أيضا أن الحساسية تنمو عبر الطبقات مع زيادة حبيبات الاضطراب. أخيرا وليس آخرا، نعرض أن النماذج بالكاد تستخدم المعلومات الموضعية لتحفيز الأشجار النحوية من تمثيلها الذاتي المتوسطة والتعويضات السياقية.
Recent research has adopted a new experimental field centered around the concept of text perturbations which has revealed that shuffled word order has little to no impact on the downstream performance of Transformer-based language models across many NLP tasks. These findings contradict the common understanding of how the models encode hierarchical and structural information and even question if the word order is modeled with position embeddings. To this end, this paper proposes nine probing datasets organized by the type of controllable text perturbation for three Indo-European languages with a varying degree of word order flexibility: English, Swedish and Russian. Based on the probing analysis of the M-BERT and M-BART models, we report that the syntactic sensitivity depends on the language and model pre-training objectives. We also find that the sensitivity grows across layers together with the increase of the perturbation granularity. Last but not least, we show that the models barely use the positional information to induce syntactic trees from their intermediate self-attention and contextualized representations.
References used
https://aclanthology.org/
rate research
Read More
Recently, a large pre-trained language model called T5 (A Unified Text-to-Text Transfer Transformer) has achieved state-of-the-art performance in many NLP tasks. However, no study has been found using this pre-trained model on Text Simplification. Th
erefore in this paper, we explore the use of T5 fine-tuning on Text Simplification combining with a controllable mechanism to regulate the system outputs that can help generate adapted text for different target audiences. Our experiments show that our model achieves remarkable results with gains of between +0.69 and +1.41 over the current state-of-the-art (BART+ACCESS). We argue that using a pre-trained model such as T5, trained on several tasks with large amounts of data, can help improve Text Simplification.
This paper describes the HEL-LJU submissions to the MultiLexNorm shared task on multilingual lexical normalization. Our system is based on a BERT token classification preprocessing step, where for each token the type of the necessary transformation i
s predicted (none, uppercase, lowercase, capitalize, modify), and a character-level SMT step where the text is translated from original to normalized given the BERT-predicted transformation constraints. For some languages, depending on the results on development data, the training data was extended by back-translating OpenSubtitles data. In the final ordering of the ten participating teams, the HEL-LJU team has taken the second place, scoring better than the previous state-of-the-art.
Pre-trained multilingual language models have become an important building block in multilingual Natural Language Processing. In the present paper, we investigate a range of such models to find out how well they transfer discourse-level knowledge acr
oss languages. This is done with a systematic evaluation on a broader set of discourse-level tasks than has been previously been assembled. We find that the XLM-RoBERTa family of models consistently show the best performance, by simultaneously being good monolingual models and degrading relatively little in a zero-shot setting. Our results also indicate that model distillation may hurt the ability of cross-lingual transfer of sentence representations, while language dissimilarity at most has a modest effect. We hope that our test suite, covering 5 tasks with a total of 22 languages in 10 distinct families, will serve as a useful evaluation platform for multilingual performance at and beyond the sentence level.
The outstanding performance of transformer-based language models on a great variety of NLP and NLU tasks has stimulated interest in exploration of their inner workings. Recent research has been primarily focused on higher-level and complex linguistic
phenomena such as syntax, semantics, world knowledge and common-sense. The majority of the studies is anglocentric, and little remains known regarding other languages, specifically their morphosyntactic properties. To this end, our work presents Morph Call, a suite of 46 probing tasks for four Indo-European languages of different morphology: Russian, French, English and German. We propose a new type of probing tasks based on detection of guided sentence perturbations. We use a combination of neuron-, layer- and representation-level introspection techniques to analyze the morphosyntactic content of four multilingual transformers, including their understudied distilled versions. Besides, we examine how fine-tuning on POS-tagging task affects the probing performance.
We study multilingual AMR parsing from the perspective of knowledge distillation, where the aim is to learn and improve a multilingual AMR parser by using an existing English parser as its teacher. We constrain our exploration in a strict multilingua
l setting: there is but one model to parse all different languages including English. We identify that noisy input and precise output are the key to successful distillation. Together with extensive pre-training, we obtain an AMR parser whose performances surpass all previously published results on four different foreign languages, including German, Spanish, Italian, and Chinese, by large margins (up to 18.8 Smatch points on Chinese and on average 11.3 Smatch points). Our parser also achieves comparable performance on English to the latest state-of-the-art English-only parser.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
