تعدد اللغات T5 Pretrains نموذج تسلسل إلى تسلسل على نصوص أحادية الأبعاد ضخمة، والتي أظهرت نتائج واعدة على العديد من المهام المتبقية اللغوية.في هذه الورقة، نحسن محول نقل النص إلى النص متعدد اللغات مع أزواج الترجمة (MT6).على وجه التحديد، نستكشف ثلاثة مهام ما قبل التدريب النصي عبر اللغات، وهي الترجمة الآلية، والفساد زوج الترجمة، وتمضم الفساد المشترك.بالإضافة إلى ذلك، نقترح هدف جزئيا غير التلقائي للتدريب المسبق للنص.نقيم الأساليب على سبع مجموعات بيانات معيار متعددة اللغات، بما في ذلك تصنيف الجملة، والاعتراف بالكياء المسمى، والإجابة على الأسئلة، والتلخيص الجماعي.تظهر النتائج التجريبية أن MT6 المقترح يحسن عملية النقل عبر اللغات عبر MT5.
Multilingual T5 pretrains a sequence-to-sequence model on massive monolingual texts, which has shown promising results on many cross-lingual tasks. In this paper, we improve multilingual text-to-text transfer Transformer with translation pairs (mT6). Specifically, we explore three cross-lingual text-to-text pre-training tasks, namely, machine translation, translation pair span corruption, and translation span corruption. In addition, we propose a partially non-autoregressive objective for text-to-text pre-training. We evaluate the methods on seven multilingual benchmark datasets, including sentence classification, named entity recognition, question answering, and abstractive summarization. Experimental results show that the proposed mT6 improves cross-lingual transferability over mT5.
References used
https://aclanthology.org/
The recent Text-to-Text Transfer Transformer'' (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 th
Recently, a large pre-trained language model called T5 (A Unified Text-to-Text Transfer Transformer) has achieved state-of-the-art performance in many NLP tasks. However, no study has been found using this pre-trained model on Text Simplification. Th
We present CoTexT, a pre-trained, transformer-based encoder-decoder model that learns the representative context between natural language (NL) and programming language (PL). Using self-supervision, CoTexT is pre-trained on large programming language
Despite the recent advances in applying pre-trained language models to generate high-quality texts, generating long passages that maintain long-range coherence is yet challenging for these models. In this paper, we propose DiscoDVT, a discourse-aware
Our increasing reliance on mobile applications means much of our communication is mediated with the support of predictive text systems. How do these systems impact interpersonal communication and broader society? In what ways are predictive text syst