Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userRethinking Network Pruning -- under the Pre-train and Fine-tune Paradigm
إعادة التفكير في التقليم الشبكة - تحت القطار ما قبل القطار والاستحقاق
274
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تحسنت نماذج اللغة المدربة مسبقا للمحولات بشكل كبير أداء مختلف مهام معالجة اللغة الطبيعية (NLP) في السنوات الأخيرة. في حين أن هذه النماذج فعالة وسائدة، فإن هذه النماذج عادة ما تكون كبيرة كبيرة لسيناريوهات النشر المحدودة للمورد. وهكذا تعمل مؤشر ترابط البحث على تطبيق تقنيات تشذيب الشبكة بموجب النموذج المؤمني ثم Finetune الذي تم تبنيه على نطاق واسع في NLP. ومع ذلك، فإن نتائج التذكيرية الحالية على المحولات القياسية، مثل بيرت، ليست ملحوظة مثل النتائج التقليم في أدب الشبكات العصبية التنافعية (CNNS). على وجه الخصوص، تنص الحكمة الشائعة في تشذيم CNN على أن تقنية تشذيب متناثرة يضغط نموذجا أكثر من ذلك الذي تم الحصول عليه عن طريق تقليل عدد القنوات والطبقات، بينما تعمل الأعمال الموجودة على تشذيب متفرق من بيرت نتائج أدنى من نظرائها الصغير الكثيف مثل Tinybert. في هذا العمل، نهدف إلى ملء هذه الفجوة من خلال دراسة كيفية نقل المعرفة وفقدت خلال عملية القطار المسبق واللغمة الدقيقة، واقتراح عملية تشذيب باري المعرفة التي تحقق نتائج متفوقة بشكل كبير من الأدب الموجود. نعرض لأول مرة أن تشذيب متفرق يضغط نموذج بيرت أكثر بكثير من تقليل عدد القنوات والطبقات. تشير التجارب في مجموعات بيانات متعددة من مؤشر الغراء إلى أن أسلوبنا تتفوق على المنافسين الرائدين مع ضغط وزن / يتخبط 20 مرة وفقدان مهملا في دقة التنبؤ.
Transformer-based pre-trained language models have significantly improved the performance of various natural language processing (NLP) tasks in the recent years. While effective and prevalent, these models are usually prohibitively large for resource-limited deployment scenarios. A thread of research has thus been working on applying network pruning techniques under the pretrain-then-finetune paradigm widely adopted in NLP. However, the existing pruning results on benchmark transformers, such as BERT, are not as remarkable as the pruning results in the literature of convolutional neural networks (CNNs). In particular, common wisdom in pruning CNN states that sparse pruning technique compresses a model more than that obtained by reducing number of channels and layers, while existing works on sparse pruning of BERT yields inferior results than its small-dense counterparts such as TinyBERT. In this work, we aim to fill this gap by studying how knowledge are transferred and lost during the pre-train, fine-tune, and pruning process, and proposing a knowledge-aware sparse pruning process that achieves significantly superior results than existing literature. We show for the first time that sparse pruning compresses a BERT model significantly more than reducing its number of channels and layers. Experiments on multiple data sets of GLUE benchmark show that our method outperforms the leading competitors with a 20-times weight/FLOPs compression and neglectable loss in prediction accuracy.
References used
https://aclanthology.org/
rate research
Read More
Recent work has demonstrated that pre-training in-domain language models can boost performance when adapting to a new domain. However, the costs associated with pre-training raise an important question: given a fixed budget, what steps should an NLP
practitioner take to maximize performance? In this paper, we study domain adaptation under budget constraints, and approach it as a customer choice problem between data annotation and pre-training. Specifically, we measure the annotation cost of three procedural text datasets and the pre-training cost of three in-domain language models. Then we evaluate the utility of different combinations of pre-training and data annotation under varying budget constraints to assess which combination strategy works best. We find that, for small budgets, spending all funds on annotation leads to the best performance; once the budget becomes large enough, a combination of data annotation and in-domain pre-training works more optimally. We therefore suggest that task-specific data annotation should be part of an economical strategy when adapting an NLP model to a new domain.
Supplementary Training on Intermediate Labeled-data Tasks (STILT) is a widely applied technique, which first fine-tunes the pretrained language models on an intermediate task before on the target task of interest. While STILT is able to further impro
ve the performance of pretrained language models, it is still unclear why and when it works. Previous research shows that those intermediate tasks involving complex inference, such as commonsense reasoning, work especially well for RoBERTa-large. In this paper, we discover that the improvement from an intermediate task could be orthogonal to it containing reasoning or other complex skills --- a simple real-fake discrimination task synthesized by GPT2 can benefit diverse target tasks. We conduct extensive experiments to study the impact of different factors on STILT. These findings suggest rethinking the role of intermediate fine-tuning in the STILT pipeline.
We introduce Dynabench, an open-source platform for dynamic dataset creation and model benchmarking. Dynabench runs in a web browser and supports human-and-model-in-the-loop dataset creation: annotators seek to create examples that a target model wil
l misclassify, but that another person will not. In this paper, we argue that Dynabench addresses a critical need in our community: contemporary models quickly achieve outstanding performance on benchmark tasks but nonetheless fail on simple challenge examples and falter in real-world scenarios. With Dynabench, dataset creation, model development, and model assessment can directly inform each other, leading to more robust and informative benchmarks. We report on four initial NLP tasks, illustrating these concepts and highlighting the promise of the platform, and address potential objections to dynamic benchmarking as a new standard for the field.
Transformer is an attention-based neural network, which consists of two sublayers, namely, Self-Attention Network (SAN) and Feed-Forward Network (FFN). Existing research explores to enhance the two sublayers separately to improve the capability of Tr
ansformer for text representation. In this paper, we present a novel understanding of SAN and FFN as Mask Attention Networks (MANs) and show that they are two special cases of MANs with static mask matrices. However, their static mask matrices limit the capability for localness modeling in text representation learning. We therefore introduce a new layer named dynamic mask attention network (DMAN) with a learnable mask matrix which is able to model localness adaptively. To incorporate advantages of DMAN, SAN, and FFN, we propose a sequential layered structure to combine the three types of layers. Extensive experiments on various tasks, including neural machine translation and text summarization demonstrate that our model outperforms the original Transformer.
We often use perturbations to regularize neural models. For neural encoder-decoders, previous studies applied the scheduled sampling (Bengio et al., 2015) and adversarial perturbations (Sato et al., 2019) as perturbations but these methods require co
nsiderable computational time. Thus, this study addresses the question of whether these approaches are efficient enough for training time. We compare several perturbations in sequence-to-sequence problems with respect to computational time. Experimental results show that the simple techniques such as word dropout (Gal and Ghahramani, 2016) and random replacement of input tokens achieve comparable (or better) scores to the recently proposed perturbations, even though these simple methods are faster.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
