Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userData Efficient Masked Language Modeling for Vision and Language
كفاءة البيانات مصممة لغة ملثمين للرؤية واللغة
627
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
النمذجة اللغوية المعقدة (MLM) هي واحدة من المهام الفرعية الرئيسية في محاكاة لغة الرؤية. في الإعداد عبر الوسائط، يتم ملثمين الرموز في الجملة بشكل عشوائي، والنموذج يتوقع أن تكون الرموز الممكنة التي أعطتها الصورة والنص. في هذه الورقة، نلاحظ العديد من عيوب MLM الرئيسية في هذا الإعداد. أولا، حيث تميل التسميات التوضيحية إلى أن تكون قصيرة، في ثلث الجمل لا يتم أخذ عينات من الجمل. ثانيا، غالبية الرموز الملثمين هي التوقف عن الكلمات وعلامات الترقيم، مما يؤدي إلى نقص في الاستخدام للصورة. إننا نحقق في مجموعة من استراتيجيات التقنيع البديلة المحددة لإعداد العرض المتعلق الذي يتناول هذه أوجه القصور، تهدف إلى توصيف أفضل من النص والصورة في التمثيل المستفاد. عند تدريب ما قبل التدريب على نموذج LXMERT، تتحسن استراتيجياتنا البديلة الخاصة بنا باستمرار عبر استراتيجية التقنيع الأصلية على ثلاثة مهام أسفل المصب، خاصة في إعدادات الموارد المنخفضة. علاوة على ذلك، يتفوق نهجنا قبل التدريب بشكل كبير على نموذج الأساس في مهمة التحقيق الفورية المصممة لاستنباط كائنات الصورة. تشير هذه النتائج وتحليلنا إلى أن طريقتنا تسمح باستفادة أفضل من بيانات التدريب.
Masked language modeling (MLM) is one of the key sub-tasks in vision-language pretraining. In the cross-modal setting, tokens in the sentence are masked at random, and the model predicts the masked tokens given the image and the text. In this paper, we observe several key disadvantages of MLM in this setting. First, as captions tend to be short, in a third of the sentences no token is sampled. Second, the majority of masked tokens are stop-words and punctuation, leading to under-utilization of the image. We investigate a range of alternative masking strategies specific to the cross-modal setting that address these shortcomings, aiming for better fusion of text and image in the learned representation. When pre-training the LXMERT model, our alternative masking strategies consistently improve over the original masking strategy on three downstream tasks, especially in low resource settings. Further, our pre-training approach substantially outperforms the baseline model on a prompt-based probing task designed to elicit image objects. These results and our analysis indicate that our method allows for better utilization of the training data.
References used
https://aclanthology.org/
rate research
Read More
An exciting frontier in natural language understanding (NLU) and generation (NLG) calls for (vision-and-) language models that can efficiently access external structured knowledge repositories. However, many existing knowledge bases only cover limite
d domains, or suffer from noisy data, and most of all are typically hard to integrate into neural language pipelines. To fill this gap, we release VisualSem: a high-quality knowledge graph (KG) which includes nodes with multilingual glosses, multiple illustrative images, and visually relevant relations. We also release a neural multi-modal retrieval model that can use images or sentences as inputs and retrieves entities in the KG. This multi-modal retrieval model can be integrated into any (neural network) model pipeline. We encourage the research community to use VisualSem for data augmentation and/or as a source of grounding, among other possible uses. VisualSem as well as the multi-modal retrieval models are publicly available and can be downloaded in this URL: https://github.com/iacercalixto/visualsem.
In the Vision-and-Language Navigation (VLN) task an embodied agent navigates a 3D environment, following natural language instructions. A challenge in this task is how to handle off the path' scenarios where an agent veers from a reference path. Prio
r work supervises the agent with actions based on the shortest path from the agent's location to the goal, but such goal-oriented supervision is often not in alignment with the instruction. Furthermore, the evaluation metrics employed by prior work do not measure how much of a language instruction the agent is able to follow. In this work, we propose a simple and effective language-aligned supervision scheme, and a new metric that measures the number of sub-instructions the agent has completed during navigation.
Detecting stance on Twitter is especially challenging because of the short length of each tweet, the continuous coinage of new terminology and hashtags, and the deviation of sentence structure from standard prose. Fine-tuned language models using lar
ge-scale in-domain data have been shown to be the new state-of-the-art for many NLP tasks, including stance detection. In this paper, we propose a novel BERT-based fine-tuning method that enhances the masked language model for stance detection. Instead of random token masking, we propose using a weighted log-odds-ratio to identify words with high stance distinguishability and then model an attention mechanism that focuses on these words. We show that our proposed approach outperforms the state of the art for stance detection on Twitter data about the 2020 US Presidential election.
We propose the task of automatically generating commentaries for races in a motor racing game, from vision, structured numerical, and textual data. Commentaries provide information to support spectators in understanding events in races. Commentary ge
neration models need to interpret the race situation and generate the correct content at the right moment. We divide the task into two subtasks: utterance timing identification and utterance generation. Because existing datasets do not have such alignments of data in multiple modalities, this setting has not been explored in depth. In this study, we introduce a new large-scale dataset that contains aligned video data, structured numerical data, and transcribed commentaries that consist of 129,226 utterances in 1,389 races in a game. Our analysis reveals that the characteristics of commentaries change over time or from viewpoints. Our experiments on the subtasks show that it is still challenging for a state-of-the-art vision encoder to capture useful information from videos to generate accurate commentaries. We make the dataset and baseline implementation publicly available for further research.
Sign language translation (SLT) is often decomposed into video-to-gloss recognition and gloss to-text translation, where a gloss is a sequence of transcribed spoken-language words in the order in which they are signed. We focus here on gloss-to-text
translation, which we treat as a low-resource neural machine translation (NMT) problem. However, unlike traditional low resource NMT, gloss-to-text translation differs because gloss-text pairs often have a higher lexical overlap and lower syntactic overlap than pairs of spoken languages. We exploit this lexical overlap and handle syntactic divergence by proposing two rule-based heuristics that generate pseudo-parallel gloss-text pairs from monolingual spoken language text. By pre-training on this synthetic data, we improve translation from American Sign Language (ASL) to English and German Sign Language (DGS) to German by up to 3.14 and 2.20 BLEU, respectively.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
