Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userLightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
LightNingDot: ما قبل التدريب تضمينات الفلالات المرئية لاسترجاع نص الصورة في الوقت الحقيقي
748
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
دفعت التدريب المسبق متعدد الوسائط إلى التقدم الرائع في أبحاث الرؤية واللغة. هذه النماذج المدربة مسبقا واسعة النطاق، على الرغم من نجاحها، تعاني مصححة من سرعة الاستدلال البطيء بسبب التكلفة الحسابية الهائلة بشكل أساسي من الاهتمام عبر الوسائط في بنية محول. عند تطبيقها على تطبيقات الحياة الحقيقية، فإن طلب الكمون والحساب الحساب يردع بشدة الاستخدام العملي للنماذج المدربة مسبقا. في هذه الورقة، ندرس استرجاع نص الصورة (ITR)، سيناريو أكثر نضجا من تطبيق V + L، الذي تمت دراسته على نطاق واسع حتى قبل ظهور النماذج المدربة مسبقا مؤخرا. نقترح نهج بسيط ولكنه فعال للغاية، الذي يسرع وقت الاستدلال في ITR بآلاف المرات، دون التضحية بالدقة. يزيل LightNingdot الاهتمام المتعلق بالعشرات المستهلكة للوقت من خلال استخراج فهارس ميزة ذات مخزيرة مؤقتا في وضع عدم الاتصال، وتوظيف مطابقة منتجات DOT الفورية عبر الإنترنت، والتي تسرع بشكل كبير عملية الاسترجاع بشكل كبير. في الواقع، يحقق LightNingDot أداء فائقا عبر معايير ITR الرئيسية مثل DataSets Flickr30k و Coco، مما يتفوق على النماذج الموجودة المدربة مسبقا تستهلك 1000 مرة من الساعات الحاسوبية باستخدام نفس الميزات.
Multimodal pre-training has propelled great advancement in vision-and-language research. These large-scale pre-trained models, although successful, fatefully suffer from slow inference speed due to enormous computational cost mainly from cross-modal attention in Transformer architecture. When applied to real-life applications, such latency and computation demand severely deter the practical use of pre-trained models. In this paper, we study Image-text retrieval (ITR), the most mature scenario of V+L application, which has been widely studied even prior to the emergence of recent pre-trained models. We propose a simple yet highly effective approach, LightningDOT that accelerates the inference time of ITR by thousands of times, without sacrificing accuracy. LightningDOT removes the time-consuming cross-modal attention by extracting pre-cached feature indexes offline, and employing instant dot-product matching online, which significantly speeds up retrieval process. In fact, our LightningDOT achieves superior performance across mainstream ITR benchmarks such as Flickr30k and COCO datasets, outperforming existing pre-trained models that consume 1000 times magnitude of computational hours using the same features.
References used
https://aclanthology.org/
rate research
Read More
Pre-trained Transformer language models (LM) have become go-to text representation encoders. Prior research fine-tunes deep LMs to encode text sequences such as sentences and passages into single dense vector representations for efficient text compar
ison and retrieval. However, dense encoders require a lot of data and sophisticated techniques to effectively train and suffer in low data situations. This paper finds a key reason is that standard LMs' internal attention structure is not ready-to-use for dense encoders, which needs to aggregate text information into the dense representation. We propose to pre-train towards dense encoder with a novel Transformer architecture, Condenser, where LM prediction CONditions on DENSE Representation. Our experiments show Condenser improves over standard LM by large margins on various text retrieval and similarity tasks.
We present VideoCLIP, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks. VideoCLIP trains a transformer for video and text by contrasting temporally overlappin
g positive video-text pairs with hard negatives from nearest neighbor retrieval. Our experiments on a diverse series of downstream tasks, including sequence-level text-video retrieval, VideoQA, token-level action localization, and action segmentation reveal state-of-the-art performance, surpassing prior work, and in some cases even outperforming supervised approaches. Code is made available at https://github.com/pytorch/fairseq/examples/MMPT.
Advances in English language representation enabled a more sample-efficient pre-training task by Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA). Which, instead of training a model to recover masked tokens, it
trains a discriminator model to distinguish true input tokens from corrupted tokens that were replaced by a generator network. On the other hand, current Arabic language representation approaches rely only on pretraining via masked language modeling. In this paper, we develop an Arabic language representation model, which we name AraELECTRA. Our model is pretrained using the replaced token detection objective on large Arabic text corpora. We evaluate our model on multiple Arabic NLP tasks, including reading comprehension, sentiment analysis, and named-entity recognition and we show that AraELECTRA outperforms current state-of-the-art Arabic language representation models, given the same pretraining data and with even a smaller model size.
Reading order detection is the cornerstone to understanding visually-rich documents (e.g., receipts and forms). Unfortunately, no existing work took advantage of advanced deep learning models because it is too laborious to annotate a large enough dat
aset. We observe that the reading order of WORD documents is embedded in their XML metadata; meanwhile, it is easy to convert WORD documents to PDFs or images. Therefore, in an automated manner, we construct ReadingBank, a benchmark dataset that contains reading order, text, and layout information for 500,000 document images covering a wide spectrum of document types. This first-ever large-scale dataset unleashes the power of deep neural networks for reading order detection. Specifically, our proposed LayoutReader captures the text and layout information for reading order prediction using the seq2seq model. It performs almost perfectly in reading order detection and significantly improves both open-source and commercial OCR engines in ordering text lines in their results in our experiments. The dataset and models are publicly available at https://aka.ms/layoutreader.
Code summarization and generation empower conversion between programming language (PL) and natural language (NL), while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, a sequence-to-sequence
model capable of performing a broad spectrum of program and language understanding and generation tasks. PLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding. Experiments on code summarization in the English language, code generation, and code translation in seven programming languages show that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program repair, clone detection, and vulnerable code detection, demonstrate PLBART's effectiveness in program understanding. Furthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow (e.g., if block inside an else block is equivalent to else if block) that are crucial to program semantics and thus excels even with limited annotations.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
