Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userPre-training a BERT with Curriculum Learning by Increasing Block-Size of Input Text
قبل التدريب برت مع التعلم من المناهج الدراسية عن طريق زيادة حجم كتلة المدخلات
329
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
حققت نماذج تمثيل اللغة المدربة مؤخرا مثل بيرت وروبرتا نتائج مهمة في مجموعة واسعة من مهام معالجة اللغة الطبيعية (NLP)، ومع ذلك، فإنها تتطلب تكلفة حسابية عالية للغاية.يعد تعلم المناهج الدراسية (CL) أحد الحلول المحتملة لتخفيف هذه المشكلة.CL هي استراتيجية تدريبية حيث يتم إعطاء عينات التدريب للنماذج بأمر هادف بدلا من أخذ العينات العشوائية.في هذا العمل، نقترح طريقة CL جديدة تدريجيا، مما يزيد تدريجيا من حجم نص المدخلات لتدريب آلية الانتباه الذاتي في بيرت ومتغيراته باستخدام الحد الأقصى للحجم الدفعة المتوفرة.تظهر التجارب في إعدادات الموارد المنخفضة أن نهجنا يفوق أن يؤدي إلى خط الأساس من حيث سرعة التقارب والأداء النهائي على مهام المصب.
Recently, pre-trained language representation models such as BERT and RoBERTa have achieved significant results in a wide range of natural language processing (NLP) tasks, however, it requires extremely high computational cost. Curriculum Learning (CL) is one of the potential solutions to alleviate this problem. CL is a training strategy where training samples are given to models in a meaningful order instead of random sampling. In this work, we propose a new CL method which gradually increases the block-size of input text for training the self-attention mechanism of BERT and its variants using the maximum available batch-size. Experiments in low-resource settings show that our approach outperforms the baseline in terms of convergence speed and final performance on downstream tasks.
References used
https://aclanthology.org/
rate research
Read More
This paper investigates and reveals the relationship between two closely related machine learning disciplines, namely Active Learning (AL) and Curriculum Learning (CL), from the lens of several novel curricula. This paper also introduces Active Curri
culum Learning (ACL) which improves AL by combining AL with CL to benefit from the dynamic nature of the AL informativeness concept as well as the human insights used in the design of the curriculum heuristics. Comparison of the performance of ACL and AL on two public datasets for the Named Entity Recognition (NER) task shows the effectiveness of combining AL and CL using our proposed framework.
Recent researches show that pre-trained models (PTMs) are beneficial to Chinese Word Segmentation (CWS). However, PTMs used in previous works usually adopt language modeling as pre-training tasks, lacking task-specific prior segmentation knowledge an
d ignoring the discrepancy between pre-training tasks and downstream CWS tasks. In this paper, we propose a CWS-specific pre-trained model MetaSeg, which employs a unified architecture and incorporates meta learning algorithm into a multi-criteria pre-training task. Empirical results show that MetaSeg could utilize common prior segmentation knowledge from different existing criteria and alleviate the discrepancy between pre-trained models and downstream CWS tasks. Besides, MetaSeg can achieve new state-of-the-art performance on twelve widely-used CWS datasets and significantly improve model performance in low-resource settings.
We take an IDEA Algorithm and add to it some stages depend on
BBM to get an Enhanced Algorithm, which had 3keys, 128-bit
input block.
In this work, we focus on a more challenging few-shot intent detection scenario where many intents are fine-grained and semantically similar. We present a simple yet effective few-shot intent detection schema via contrastive pre-training and fine-tun
ing. Specifically, we first conduct self-supervised contrastive pre-training on collected intent datasets, which implicitly learns to discriminate semantically similar utterances without using any labels. We then perform few-shot intent detection together with supervised contrastive learning, which explicitly pulls utterances from the same intent closer and pushes utterances across different intents farther. Experimental results show that our proposed method achieves state-of-the-art performance on three challenging intent detection datasets under 5-shot and 10-shot settings.
We address the compositionality challenge presented by the SCAN benchmark. Using data augmentation and a modification of the standard seq2seq architecture with attention, we achieve SOTA results on all the relevant tasks from the benchmark, showing t
he models can generalize to words used in unseen contexts. We propose an extension of the benchmark by a harder task, which cannot be solved by the proposed method.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
