Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userInfusing Finetuning with Semantic Dependencies
infusing finetuning مع التبعيات الدلالية
192
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
مجردة لأنظمة معالجة اللغات الطبيعية، وهي نوعين من الأدلة تدعم استخدام تمثيلات نصية من نماذج اللغة العصبية المحددة "على الفوروريا غير المخلفات الكبيرة: الأداء على معايير مستوحاة من التطبيقات (بيترز وآخرون، 2018، في جملة أمور أخرى)، والظهور من التجريدات النحوية في تلك التمثيلات (Tenney et al.، 2019، في جملة أمور أخرى). من ناحية أخرى، فإن الافتقار إلى الإشراف الأساسي يدعو إلى المسائل مدى جودة هذه التمثيلات يمكن أن تلتقط المعنى (Bender and Koller، 2020). نحن نطبق تحقيقات جديدة إلى نماذج اللغة الأخيرة --- التركيز على وجه التحديد على هيكل الوسائد المسند على النحو الذي يتعرض عليه التبعيات الدلالية (إيفانوفا وآخرون، 2012) --- وإيجاد ذلك، على عكس بناء الجملة، لا يتم إحضار الدلالات إلى السطح من قبل اليوم نماذج مسبقا. بعد ذلك، نستخدم تشفير الرسومات التنافيلية لتشمل صراحة على التقييم الدلالي في الفوائد الخاصة بمهام المهام، وتحقيق فوائد العائد لمهام فهم اللغة الطبيعية (NLU) في معيار الغراء. يوضح هذا النهج إمكانية الإشراف اللغوي للأغراض العامة (بدلا من المهام الخاصة)، أعلاه وما يتجاوز الاحتجاج والتأمل التقليدي. تساعد العديد من التشخيص في توطين فوائد نهجنا
Abstract For natural language processing systems, two kinds of evidence support the use of text representations from neural language models pretrained'' on large unannotated corpora: performance on application-inspired benchmarks (Peters et al., 2018, inter alia), and the emergence of syntactic abstractions in those representations (Tenney et al., 2019, inter alia). On the other hand, the lack of grounded supervision calls into question how well these representations can ever capture meaning (Bender and Koller, 2020). We apply novel probes to recent language models--- specifically focusing on predicate-argument structure as operationalized by semantic dependencies (Ivanova et al., 2012)---and find that, unlike syntax, semantics is not brought to the surface by today's pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning, yielding benefits to natural language understanding (NLU) tasks in the GLUE benchmark. This approach demonstrates the potential for general-purpose (rather than task-specific) linguistic supervision, above and beyond conventional pretraining and finetuning. Several diagnostics help to localize the benefits of our approach.1
References used
https://aclanthology.org/
rate research
Read More
We propose pre-finetuning, an additional large-scale learning stage between language model pre-training and fine-tuning. Pre-finetuning is massively multi-task learning (around 50 datasets, over 4.8 million total labeled examples), and is designed to
encourage learning of representations that generalize better to many different tasks. We show that pre-finetuning consistently improves performance for pretrained discriminators (e.g. RoBERTa) and generation models (e.g. BART) on a wide range of tasks (sentence prediction, commonsense reasoning, MRC, etc.), while also significantly improving sample efficiency during fine-tuning. We also show that large-scale multi-tasking is crucial; pre-finetuning can hurt performance when few tasks are used up until a critical point (usually above 15) after which performance improves linearly in the number of tasks.
This paper details experiments we performed on the Universal Dependencies 2.7 corpora in order to investigate the dominant word order in the available languages. For this purpose, we used a graph rewriting tool, GREW, which allowed us to go beyond th
e surface annotations and identify the implicit subjects. We first measured the distribution of the six different word orders (SVO, SOV, VSO, VOS, OVS, OSV) in the corpora and investigated when there was a significant difference in the corpora within a given language. Then, we compared the obtained results with information provided in the WALS database (Dryer and Haspelmath, 2013) and in ( ̈Ostling, 2015). Finally, we examined the impact of using a graph rewriting tool for this task. The tools and resources used for this research are all freely available.
Fully understanding narratives often requires identifying events in the context of whole documents and modeling the event relations. However, document-level event extraction is a challenging task as it requires the extraction of event and entity core
ference, and capturing arguments that span across different sentences. Existing works on event extraction usually confine on extracting events from single sentences, which fail to capture the relationships between the event mentions at the scale of a document, as well as the event arguments that appear in a different sentence than the event trigger. In this paper, we propose an end-to-end model leveraging Deep Value Networks (DVN), a structured prediction algorithm, to efficiently capture cross-event dependencies for document-level event extraction. Experimental results show that our approach achieves comparable performance to CRF-based models on ACE05, while enjoys significantly higher computational efficiency.
Text variational autoencoders (VAEs) are notorious for posterior collapse, a phenomenon where the model's decoder learns to ignore signals from the encoder. Because posterior collapse is known to be exacerbated by expressive decoders, Transformers ha
ve seen limited adoption as components of text VAEs. Existing studies that incorporate Transformers into text VAEs (Li et al., 2020; Fang et al., 2021) mitigate posterior collapse using massive pretraining, a technique unavailable to most of the research community without extensive computing resources. We present a simple two-phase training scheme to convert a sequence-to-sequence Transformer into a VAE with just finetuning. The resulting language model is competitive with massively pretrained Transformer-based VAEs in some internal metrics while falling short on others. To facilitate training we comprehensively explore the impact of common posterior collapse alleviation techniques in the literature. We release our code for reproducability.
Video Question Answering (VidQA) evaluation metrics have been limited to a single-word answer or selecting a phrase from a fixed set of phrases. These metrics limit the VidQA models' application scenario. In this work, we leverage semantic roles deri
ved from video descriptions to mask out certain phrases, to introduce VidQAP which poses VidQA as a fill-in-the-phrase task. To enable evaluation of answer phrases, we compute the relative improvement of the predicted answer compared to an empty string. To reduce the influence of language bias in VidQA datasets, we retrieve a video having a different answer for the same question. To facilitate research, we construct ActivityNet-SRL-QA and Charades-SRL-QA and benchmark them by extending three vision-language models. We perform extensive analysis and ablative studies to guide future work. Code and data are public.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
