أصبحت نماذج لغة كبيرة من الصعب تدريبا على نحو متزايد بسبب وقت الحسبان المتزايد والتكلفة.في هذا العمل، نقدم SRU ++، وهي عبارة عن بنية عالية الكفاءة تجمع بين تكرار سريع واهتمام لنمذجة التسلسل.SRU ++ يعرض قدرة النمذجة القوية وكفاءة التدريب.فيما يتعلق بمهام نمذجة اللغة القياسية مثل مجموعات بيانات ENWIK8 و Wiki-103 و Mount Word Word، يحصل نموذجنا على أجزاء أفضل لكل حرف وحيرة أثناء استخدام تكلفة التدريب الأقل بنسبة 3x-10x مقارنة بنماذج المحولات ذات الأداء الأعلى.على سبيل المثال، يحقق نموذجنا نتيجة حديثة لمجموعة بيانات Enwik8 باستخدام 1.6 أيام من التدريب على آلة 8 GPU.نوضح كذلك أن SRU ++ يتطلب الحد الأدنى من الاهتمام بالقرب من الأداء القريب من الحديث.تشير نتائجنا إلى الاستفادة بشكل مشترك تكرار سريع مع القليل من الاهتمام باعتباره اتجاها واعدا لتسريع التدريب النموذجي والاستدلال.
Large language models have become increasingly difficult to train because of the growing computation time and cost. In this work, we present SRU++, a highly-efficient architecture that combines fast recurrence and attention for sequence modeling. SRU++ exhibits strong modeling capacity and training efficiency. On standard language modeling tasks such as Enwik8, Wiki-103 and Billion Word datasets, our model obtains better bits-per-character and perplexity while using 3x-10x less training cost compared to top-performing Transformer models. For instance, our model achieves a state-of-the-art result on the Enwik8 dataset using 1.6 days of training on an 8-GPU machine. We further demonstrate that SRU++ requires minimal attention for near state-of-the-art performance. Our results suggest jointly leveraging fast recurrence with little attention as a promising direction for accelerating model training and inference.
References used
https://aclanthology.org/
Previous work has indicated that pretrained Masked Language Models (MLMs) are not effective as universal lexical and sentence encoders off-the-shelf, i.e., without further task-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasks u
Advertising on e-commerce and social media sites deliver ad impressions at web scale on a daily basis driving value to both shoppers and advertisers. This scale necessitates programmatic ways of detecting unsuitable content in ads to safeguard custom
Scenario-based question answering (SQA) requires retrieving and reading paragraphs from a large corpus to answer a question which is contextualized by a long scenario description. Since a scenario contains both keyphrases for retrieval and much noise
Transfer learning based on pretraining language models on a large amount of raw data has become a new norm to reach state-of-the-art performance in NLP. Still, it remains unclear how this approach should be applied for unseen languages that are not c
Large pretrained language models using the transformer neural network architecture are becoming a dominant methodology for many natural language processing tasks, such as question answering, text classification, word sense disambiguation, text comple