أصبحت نماذج لغة المحولات المدربة مسبقا (LM) لتشفيات تمثيل النص.البحث المسبق يلتزم LMS عميق لتشفير تسلسل النص مثل الجمل والمرورات في تمثيلات ناقلات كثيفة واحدة لمقارنة النص وانتبعدة فعالة.ومع ذلك، تتطلب التشفير الكثيفة الكثير من البيانات والتقنيات المتطورة للتدريب بشكل فعال وتعاني في مواقف البيانات المنخفضة.تجد هذه الورقة سبب رئيسي هو أن هيكل العناية الداخلية القياسية ل LMS غير جاهزة للاستخدام للترميزات الكثيفة، والتي تحتاج إلى إجمالي معلومات نصية في التمثيل الكثيف.نقترح ما قبل القطار نحو التشفير الكثيف مع بنية محول رواية، مكثف، حيث ظروف التنبؤ LM على تمثيل كثيف.تعرض تجاربنا تظهر المكثف يحسن أكثر من LM القياسية من قبل هوامش كبيرة على مهام استرجاع النص المختلفة والتشابه.
Pre-trained Transformer language models (LM) have become go-to text representation encoders. Prior research fine-tunes deep LMs to encode text sequences such as sentences and passages into single dense vector representations for efficient text comparison and retrieval. However, dense encoders require a lot of data and sophisticated techniques to effectively train and suffer in low data situations. This paper finds a key reason is that standard LMs' internal attention structure is not ready-to-use for dense encoders, which needs to aggregate text information into the dense representation. We propose to pre-train towards dense encoder with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. Our experiments show Condenser improves over standard LM by large margins on various text retrieval and similarity tasks.
References used
https://aclanthology.org/
Multimodal pre-training has propelled great advancement in vision-and-language research. These large-scale pre-trained models, although successful, fatefully suffer from slow inference speed due to enormous computational cost mainly from cross-modal
We present Mr. TyDi, a multi-lingual benchmark dataset for mono-lingual retrieval in eleven typologically diverse languages, designed to evaluate ranking with learned dense representations. The goal of this resource is to spur research in dense retri
Dense retrieval has shown great success for passage ranking in English. However, its effectiveness for non-English languages remains unexplored due to limitation in training resources. In this work, we explore different transfer techniques for docume
Code summarization and generation empower conversion between programming language (PL) and natural language (NL), while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, a sequence-to-sequence
Recent researches show that pre-trained models (PTMs) are beneficial to Chinese Word Segmentation (CWS). However, PTMs used in previous works usually adopt language modeling as pre-training tasks, lacking task-specific prior segmentation knowledge an