Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userStructure-aware Fine-tuning of Sequence-to-sequence Transformers for Transition-based AMR Parsing
الهيكل - يدرك ضبط محولات تسلسل إلى تسلسل لتحليل عمرو القائمة على الانتقال
395
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
أدت مؤخرا مؤخرا الرسوم البيانية تم التنبؤ بمعنى التجريدي المعني (AMR) باستخدام نماذج محولات تسلسل التسلسل المدربة مسبقا إلى تحسينات كبيرة على معايير تحليل AMR. هذه المحللون بسيطة وتجنب النمذجة الصريحة للهيكل ولكن تفتقر إلى خصائص مرغوبة مثل ضمانات الرسوم البيانية بشكل جيد أو محاذاة الرسم الرسم البياني المدمج. في هذا العمل، نستكشف دمج نماذج لغة التسلسل العامة المدربة مسبقا ونهج بناء على الهيكل. نغادر من نظام انتقال مقره مؤشر واقتراح مجموعة انتقالية مبسطة، مصممة لتحسين استغلال نماذج اللغة المدربة مسبقا للضبط الناعم الهيكل. نحن نستكشف أيضا نمذجة ولاية المحللين داخل بنية فك التشفير المدربة مسبقا واستراتيجيات المفردات المختلفة لنفس الغرض. نحن نقدم مقارنة مفصلة مع التقدم المحرز الأخير في تحليل عمرو وإظهار أن المحلل المحلل المقترح يحتفظ بالخصائص المرغوبة للمناهج السابقة القائمة على الانتقال، بينما تكون أكثر بساطة والوصول إلى حالة التحليل الجديدة للفن AMR 2.0، دون الحاجة إلى إعادة الرسم البياني الفصداء.
Predicting linearized Abstract Meaning Representation (AMR) graphs using pre-trained sequence-to-sequence Transformer models has recently led to large improvements on AMR parsing benchmarks. These parsers are simple and avoid explicit modeling of structure but lack desirable properties such as graph well-formedness guarantees or built-in graph-sentence alignments. In this work we explore the integration of general pre-trained sequence-to-sequence language models and a structure-aware transition-based approach. We depart from a pointer-based transition system and propose a simplified transition set, designed to better exploit pre-trained language models for structured fine-tuning. We also explore modeling the parser state within the pre-trained encoder-decoder architecture and different vocabulary strategies for the same purpose. We provide a detailed comparison with recent progress in AMR parsing and show that the proposed parser retains the desirable properties of previous transition-based approaches, while being simpler and reaching the new parsing state of the art for AMR 2.0, without the need for graph re-categorization.
References used
https://aclanthology.org/
rate research
Read More
Current benchmark tasks for natural language processing contain text that is qualitatively different from the text used in informal day to day digital communication. This discrepancy has led to severe performance degradation of state-of-the-art NLP m
odels when fine-tuned on real-world data. One way to resolve this issue is through lexical normalization, which is the process of transforming non-standard text, usually from social media, into a more standardized form. In this work, we propose a sentence-level sequence-to-sequence model based on mBART, which frames the problem as a machine translation problem. As the noisy text is a pervasive problem across languages, not just English, we leverage the multi-lingual pre-training of mBART to fine-tune it to our data. While current approaches mainly operate at the word or subword level, we argue that this approach is straightforward from a technical standpoint and builds upon existing pre-trained transformer networks. Our results show that while word-level, intrinsic, performance evaluation is behind other methods, our model improves performance on extrinsic, downstream tasks through normalization compared to models operating on raw, unprocessed, social media text.
We investigate how sentence-level transformers can be modified into effective sequence labelers at the token level without any direct supervision. Existing approaches to zero-shot sequence labeling do not perform well when applied on transformer-base
d architectures. As transformers contain multiple layers of multi-head self-attention, information in the sentence gets distributed between many tokens, negatively affecting zero-shot token-level performance. We find that a soft attention module which explicitly encourages sharpness of attention weights can significantly outperform existing methods.
Complex natural language applications such as speech translation or pivot translation traditionally rely on cascaded models. However,cascaded models are known to be prone to error propagation and model discrepancy problems. Furthermore, there is no p
ossibility of using end-to-end training data in conventional cascaded systems, meaning that the training data most suited for the task cannot be used.Previous studies suggested several approaches for integrated end-to-end training to overcome those problems, however they mostly rely on(synthetic or natural) three-way data. We propose a cascaded model based on the non-autoregressive Transformer that enables end-to-end training without the need for an explicit intermediate representation. This new architecture (i) avoids unnecessary early decisions that can cause errors which are then propagated throughout the cascaded models and (ii) utilizes the end-to-end training data directly. We conduct an evaluation on two pivot-based machine translation tasks, namely French→German and German→Czech. Our experimental results show that the proposed architecture yields an improvement of more than 2 BLEU for French→German over the cascaded baseline.
Non-autoregressive machine translation (NAT) approaches enable fast generation by utilizing parallelizable generative processes. The remaining bottleneck in these models is their decoder layers; unfortunately unlike in autoregressive models (Kasai et
al., 2020), removing decoder layers from NAT models significantly degrades accuracy. This work proposes a sequence-to-lattice model that replaces the decoder with a search lattice. Our approach first constructs a candidate lattice using efficient lookup operations, generates lattice scores from a deep encoder, and finally finds the best path using dynamic programming. Experiments on three machine translation datasets show that our method is faster than past non-autoregressive generation approaches, and more accurate than naively reducing the number of decoder layers.
Data-to-text (D2T) generation in the biomedical domain is a promising - yet mostly unexplored - field of research. Here, we apply neural models for D2T generation to a real-world dataset consisting of package leaflets of European medicines. We show t
hat fine-tuned transformers are able to generate realistic, multi-sentence text from data in the biomedical domain, yet have important limitations. We also release a new dataset (BioLeaflets) for benchmarking D2T generation models in the biomedical domain.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
