Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userWikiBERT Models: Deep Transfer Learning for Many Languages
نماذج Wikibert: التعلم العميق التعلم للعديد من اللغات
619
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
مكنت نماذج اللغة العصبية العميقة مثل بيرت التطورات الأخيرة في العديد من مهام معالجة اللغة الطبيعية. ومع ذلك، نظرا للجهد والتكلفة الحاسوبية المشاركة في التدريب المسبق لها، يتم إدخال هذه النماذج عادة فقط لعدد صغير من لغات الموارد عالية الوزن مثل اللغة الإنجليزية. في حين تتوفر نماذج متعددة اللغات التي تغطي أعدادا كبيرة من اللغات، فإن العمل الحديث يشير إلى أن التدريب أحادي الأحادي يمكن أن ينتج عن نماذج أفضل، وفهمنا للمفاضرة بين التدريب الأحادي وغير اللغوي غير مكتمل. في هذه الورقة، نقدم خط أنابيب بسيطة وأتمتة بالكامل لإنشاء نماذج بيرت الخاصة باللغة من البيانات من بيانات ويكيبيديا وإدخال 42 من هذه النماذج الجديدة، والأكثر من اللازم لغات حتى الآن تفتقر إلى نماذج اللغة العصبية العميقة المخصصة. نقوم بتقييم مزايا هذه النماذج باستخدام اختبارات Cloze و Autify Parser على بيانات التبعيات العالمية، والأداء المتناقض مع النتائج باستخدام طراز Bert (Mbert) متعدد اللغات. نجد أن نماذج WikiBert المقدمة حديثا تفوقت Mbert في اختبارات Cloze لجميع اللغات تقريبا، وأن uDify باستخدام نماذج Wikibert تفوق المحلل باستخدام Mbert في المتوسط، مع توضح الطرز الخاصة باللغة تحسين أداء محسنة بشكل كبير لبعض اللغات، ولكن تحسين محدود أو تحسين انخفاض في الأداء للآخرين. تتوفر جميع الطرق والنماذج المقدمة في هذا العمل تحت التراخيص المفتوحة من https://github.com/turkunlp/wikibert.
Deep neural language models such as BERT have enabled substantial recent advances in many natural language processing tasks. However, due to the effort and computational cost involved in their pre-training, such models are typically introduced only for a small number of high-resource languages such as English. While multilingual models covering large numbers of languages are available, recent work suggests monolingual training can produce better models, and our understanding of the tradeoffs between mono- and multilingual training is incomplete. In this paper, we introduce a simple, fully automated pipeline for creating language-specific BERT models from Wikipedia data and introduce 42 new such models, most for languages up to now lacking dedicated deep neural language models. We assess the merits of these models using cloze tests and the state-of-the-art UDify parser on Universal Dependencies data, contrasting performance with results using the multilingual BERT (mBERT) model. We find that the newly introduced WikiBERT models outperform mBERT in cloze tests for nearly all languages, and that UDify using WikiBERT models outperforms the parser using mBERT on average, with the language-specific models showing substantially improved performance for some languages, yet limited improvement or a decrease in performance for others. All of the methods and models introduced in this work are available under open licenses from https://github.com/turkunlp/wikibert.
References used
https://aclanthology.org/
rate research
Read More
Active learning has been shown to reduce annotation requirements for numerous natural language processing tasks, including semantic role labeling (SRL). SRL involves labeling argument spans for potentially multiple predicates in a sentence, which mak
es it challenging to aggregate the numerous decisions into a single score for determining new instances to annotate. In this paper, we apply two ways of aggregating scores across multiple predicates in order to choose query sentences with two methods of estimating model certainty: using the neural network's outputs and using dropout-based Bayesian Active Learning by Disagreement. We compare these methods with three passive baselines --- random sentence selection, random whole-document selection, and selecting sentences with the most predicates --- and analyse the effect these strategies have on the learning curve with respect to reducing the number of annotated sentences and predicates to achieve high performance.
Deep learning is at the heart of the current rise of artificial intelligence. In the field of Computer Vision, it has become the workhorse for applications ranging from self-driving cars to surveillance and security. Whereas deep neural networks have
demonstrated phenomenal success (often beyond human capabilities) in solving complex problems, recent studies show that they are vulnerable to adversarial attacks in the form of subtle perturbations to inputs that lead a model to predict incorrect outputs. For images, such perturbations are often too small to be perceptible, yet they completely fool the deep learning models. Adversarial attacks pose a serious threat to the success of deep learning in practice. This fact has recently lead to a large influx of contributions in this direction. This article presents a survey on adversarial attacks on deep learning in Computer Vision. We review the works that design adversarial attacks, analyze the existence of such attacks and propose defenses against them
Logical Observation Identifiers Names and Codes (LOINC) is a standard set of codes that enable clinicians to communicate about medical tests. Laboratories depend on LOINC to identify what tests a doctor orders for a patient. However, clinicians often
use site specific, custom codes in their medical records systems that can include shorthand, spelling mistakes, and invented acronyms. Software solutions must map from these custom codes to the LOINC standard to support data interoperability. A key challenge is that LOINC is comprised of six elements. Mapping requires not only extracting those elements, but also combining them according to LOINC logic. We found that character-based deep learning excels at extracting LOINC elements while logic based methods are more effective for combining those elements into complete LOINC values. In this paper, we present an ensemble of machine learning and logic that is currently used in several medical facilities to map from
The exponential growth of the internet and social media in the past decade gave way to the increase in dissemination of false or misleading information. Since the 2016 US presidential election, the term fake news'' became increasingly popular and thi
s phenomenon has received more attention. In the past years several fact-checking agencies were created, but due to the great number of daily posts on social media, manual checking is insufficient. Currently, there is a pressing need for automatic fake news detection tools, either to assist manual fact-checkers or to operate as standalone tools. There are several projects underway on this topic, but most of them focus on English. This research-in-progress paper discusses the employment of deep learning methods, and the development of a tool, for detecting false news in Portuguese. As a first step we shall compare well-established architectures that were tested in other languages and analyse their performance on our Portuguese data. Based on the preliminary results of these classifiers, we shall choose a deep learning model or combine several deep learning models which hold promise to enhance the performance of our fake news detection system.
Due to its great power in modeling non-Euclidean data like graphs or manifolds, deep learning on graph techniques (i.e., Graph Neural Networks (GNNs)) have opened a new door to solving challenging graph-related NLP problems. There has seen a surge of
interests in applying deep learning on graph techniques to NLP, and has achieved considerable success in many NLP tasks, ranging from classification tasks like sentence classification, semantic role labeling and relation extraction, to generation tasks like machine translation, question generation and summarization. Despite these successes, deep learning on graphs for NLP still face many challenges, including automatically transforming original text sequence data into highly graph-structured data, and effectively modeling complex data that involves mapping between graph-based inputs and other highly structured output data such as sequences, trees, and graph data with multi-types in both nodes and edges. This tutorial will cover relevant and interesting topics on applying deep learning on graph techniques to NLP, including automatic graph construction for NLP, graph representation learning for NLP, advanced GNN based models (e.g., graph2seq, graph2tree, and graph2graph) for NLP, and the applications of GNNs in various NLP tasks (e.g., machine translation, natural language generation, information extraction and semantic parsing). In addition, hands-on demonstration sessions will be included to help the audience gain practical experience on applying GNNs to solve challenging NLP problems using our recently developed open source library -- Graph4NLP, the first library for researchers and practitioners for easy use of GNNs for various NLP tasks.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
