Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userCoRT: Complementary Rankings from Transformers
كورت: تصنيفات تكميلية من المحولات
166
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
العديد من الأساليب الأخيرة تجاه استرجاع المعلومات العصبية تخفف من تكاليفها الحاسوبية باستخدام خط أنابيب الترتيب متعدد المراحل.في المرحلة الأولى، يتم استرجاع عدد من المرشحين المحتملين ذوي الصلة باستخدام نموذج استرجاع فعال مثل BM25.على الرغم من أن BM25 قد أثبت أداء لائق كمرفاة في المرحلة الأولى، فإنه يميل إلى تفويت الممرات ذات الصلة.في هذا السياق، نقترح كورت، وهو نموذج بسيط في المرحلة الأولى من المرحلة الأولى يرفع تمثيلات سياقية من نماذج اللغة المسبقة مسبقا مثل بيرت لاستكمال وظائف الترتيب القائمة على الأجل مع عدم التسبب في عدم وجود تأخير كبير في وقت الاستعلام.باستخدام DataSet MS Marco، نظهر أن Cort يزيد بشكل كبير من استدعاء المرشح من خلال استكمال BM25 مع المرشحين المفقودين.وبالتالي، نجد أن إعادة الراهنات اللاحقة تحقيق نتائج فائقة مع أقل مرشحين.نوضح كذلك أن استرجاع المرور باستخدام CORT يمكن تحقيقه مع انخفاض الآمون المنخفض بشكل مدهش.
Many recent approaches towards neural information retrieval mitigate their computational costs by using a multi-stage ranking pipeline. In the first stage, a number of potentially relevant candidates are retrieved using an efficient retrieval model such as BM25. Although BM25 has proven decent performance as a first-stage ranker, it tends to miss relevant passages. In this context we propose CoRT, a simple neural first-stage ranking model that leverages contextual representations from pretrained language models such as BERT to complement term-based ranking functions while causing no significant delay at query time. Using the MS MARCO dataset, we show that CoRT significantly increases the candidate recall by complementing BM25 with missing candidates. Consequently, we find subsequent re-rankers achieve superior results with less candidates. We further demonstrate that passage retrieval using CoRT can be realized with surprisingly low latencies.
References used
https://aclanthology.org/
rate research
Read More
The current approach to collecting human judgments of machine translation quality for the news translation task at WMT -- segment rating with document context -- is the most recent in a sequence of changes to WMT human annotation protocol. As these a
nnotation protocols have changed over time, they have drifted away from some of the initial statistical assumptions underpinning them, with consequences that call the validity of WMT news task system rankings into question. In simulations based on real data, we show that the rankings can be influenced by the presence of outliers (high- or low-quality systems), resulting in different system rankings and clusterings. We also examine questions of annotation task composition and how ease or difficulty of translating different documents may influence system rankings. We provide discussion of ways to analyze these issues when considering future changes to annotation protocols.
In Visual Question Answering (VQA), existing bilinear methods focus on the interaction between images and questions. As a result, the answers are either spliced into the questions or utilized as labels only for classification. On the other hand, tril
inear models such as the CTI model efficiently utilize the inter-modality information between answers, questions, and images, while ignoring intra-modality information. Inspired by this observation, we propose a new trilinear interaction framework called MIRTT (Learning Multimodal Interaction Representations from Trilinear Transformers), incorporating the attention mechanisms for capturing inter-modality and intra-modality relationships. Moreover, we design a two-stage workflow where a bilinear model reduces the free-form, open-ended VQA problem into a multiple-choice VQA problem. Furthermore, to obtain accurate and generic multimodal representations, we pre-train MIRTT with masked language prediction. Our method achieves state-of-the-art performance on the Visual7W Telling task and VQA-1.0 Multiple Choice task and outperforms bilinear baselines on the VQA-2.0, TDIUC and GQA datasets.
Being able to accurately perform Question Difficulty Estimation (QDE) can improve the accuracy of students' assessment and better their learning experience. Traditional approaches to QDE are either subjective or introduce a long delay before new ques
tions can be used to assess students. Thus, recent work proposed machine learning-based approaches to overcome these limitations. They use questions of known difficulty to train models capable of inferring the difficulty of questions from their text. Once trained, they can be used to perform QDE of newly created questions. Existing approaches employ supervised models which are domain-dependent and require a large dataset of questions of known difficulty for training. Therefore, they cannot be used if such a dataset is not available ( for new courses on an e-learning platform). In this work, we experiment with the possibility of performing QDE from text in an unsupervised manner. Specifically, we use the uncertainty of calibrated question answering models as a proxy of human-perceived difficulty. Our experiments show promising results, suggesting that model uncertainty could be successfully leveraged to perform QDE from text, reducing both costs and elapsed time.
Representations from large pretrained models such as BERT encode a range of features into monolithic vectors, affording strong predictive accuracy across a range of downstream tasks. In this paper we explore whether it is possible to learn disentangl
ed representations by identifying existing subnetworks within pretrained models that encode distinct, complementary aspects. Concretely, we learn binary masks over transformer weights or hidden units to uncover subsets of features that correlate with a specific factor of variation; this eliminates the need to train a disentangled model from scratch for a particular task. We evaluate this method with respect to its ability to disentangle representations of sentiment from genre in movie reviews, toxicity from dialect in Tweets, and syntax from semantics. By combining masking with magnitude pruning we find that we can identify sparse subnetworks within BERT that strongly encode particular aspects (e.g., semantics) while only weakly encoding others (e.g., syntax). Moreover, despite only learning masks, disentanglement-via-masking performs as well as --- and often better than ---previously proposed methods based on variational autoencoders and adversarial training.
Transformer models are expensive to fine-tune, slow for inference, and have large storage requirements. Recent approaches tackle these shortcomings by training smaller models, dynamically reducing the model size, and by training light-weight adapters
. In this paper, we propose AdapterDrop, removing adapters from lower transformer layers during training and inference, which incorporates concepts from all three directions. We show that AdapterDrop can dynamically reduce the computational overhead when performing inference over multiple tasks simultaneously, with minimal decrease in task performances. We further prune adapters from AdapterFusion, which improves the inference efficiency while maintaining the task performances entirely.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
