Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userTowards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU
نحو المحولات الإضافية: تحليل تجريبي لنماذج المحولات ل NLU تدريجي
362
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
تتيح المعالجة الإضافية أنظمة تفاعلية تستجيب بناء على المدخلات الجزئية، وهي خاصية مرغوبة على سبيل المثال في عوامل الحوار. تقوم بنية المحولات الشعبية حاليا بطبيعتها بمعالجة التسلسلات ككل، تجرد فكرة الوقت. محاولات العمل الحديثة لتطبيق المحولات بشكل تدريجي عن طريق إعادة التشغيل - تدريجيا من خلال التغذية بشكل متكرر، إلى نموذج غير متقلب، بادئات إدخال أطول بشكل متزايد لإنتاج مخرجات جزئية. ومع ذلك، فإن هذا النهج مكلف بشكل حسابي ولا يتجادل بكفاءة للتسلسل الطويل. بالتوازي، نشهد جهود لجعل المحولات أكثر كفاءة، على سبيل المثال المحول الخطي (LT) مع آلية تكرار. في هذا العمل، ندرس جدوى LT ل NLU تدريجي باللغة الإنجليزية. تبين نتائجنا أن نموذج LT المتكرر لديه أفضل أداء تدريجي وسرعة الاستدلال أسرع مقارنة بالمحول القياسي واللفنت مع إعادة التشغيل التدريجي، بتكلفة جزء من جودة غير متزايدة (التسلسل الكامل). نظرا لأن إسقاط الأداء يمكن تخفيفه عن طريق تدريب النموذج لانتظار السياق الصحيح قبل الالتزام بإخراج وأن التدريب بادئة الإدخال مفيد لتقديم المخرجات الجزئية الصحيحة.
Incremental processing allows interactive systems to respond based on partial inputs, which is a desirable property e.g. in dialogue agents. The currently popular Transformer architecture inherently processes sequences as a whole, abstracting away the notion of time. Recent work attempts to apply Transformers incrementally via restart-incrementality by repeatedly feeding, to an unchanged model, increasingly longer input prefixes to produce partial outputs. However, this approach is computationally costly and does not scale efficiently for long sequences. In parallel, we witness efforts to make Transformers more efficient, e.g. the Linear Transformer (LT) with a recurrence mechanism. In this work, we examine the feasibility of LT for incremental NLU in English. Our results show that the recurrent LT model has better incremental performance and faster inference speed compared to the standard Transformer and LT with restart-incrementality, at the cost of part of the non-incremental (full sequence) quality. We show that the performance drop can be mitigated by training the model to wait for right context before committing to an output and that training with input prefixes is beneficial for delivering correct partial outputs.
References used
https://aclanthology.org/
rate research
Read More
Neural Topic Models are recent neural models that aim at extracting the main themes from a collection of documents. The comparison of these models is usually limited because the hyperparameters are held fixed. In this paper, we present an empirical a
nalysis and comparison of Neural Topic Models by finding the optimal hyperparameters of each model for four different performance measures adopting a single-objective Bayesian optimization. This allows us to determine the robustness of a topic model for several evaluation metrics. We also empirically show the effect of the length of the documents on different optimized metrics and discover which evaluation metrics are in conflict or agreement with each other.
Recent studies indicate that NLU models are prone to rely on shortcut features for prediction, without achieving true language understanding. As a result, these models fail to generalize to real-world out-of-distribution data. In this work, we show t
hat the words in the NLU training set can be modeled as a long-tailed distribution. There are two findings: 1) NLU models have strong preference for features located at the head of the long-tailed distribution, and 2) Shortcut features are picked up during very early few iterations of the model training. These two observations are further employed to formulate a measurement which can quantify the shortcut degree of each training sample. Based on this shortcut measurement, we propose a shortcut mitigation framework LGTR, to suppress the model from making overconfident predictions for samples with large shortcut degree. Experimental results on three NLU benchmarks demonstrate that our long-tailed distribution explanation accurately reflects the shortcut learning behavior of NLU models. Experimental analysis further indicates that LGTR can improve the generalization accuracy on OOD data, while preserving the accuracy on in-distribution data.
Relative position embedding (RPE) is a successful method to explicitly and efficaciously encode position information into Transformer models. In this paper, we investigate the potential problems in Shaw-RPE and XL-RPE, which are the most representati
ve and prevalent RPEs, and propose two novel RPEs called Low-level Fine-grained High-level Coarse-grained (LFHC) RPE and Gaussian Cumulative Distribution Function (GCDF) RPE. LFHC-RPE is an improvement of Shaw-RPE, which enhances the perception ability at medium and long relative positions. GCDF-RPE utilizes the excellent properties of the Gaussian function to amend the prior encoding mechanism in XL-RPE. Experimental results on nine authoritative datasets demonstrate the effectiveness of our methods empirically. Furthermore, GCDF-RPE achieves the best overall performance among five different RPEs.
Syntax is fundamental to our thinking about language. Failing to capture the structure of input language could lead to generalization problems and over-parametrization. In the present work, we propose a new syntax-aware language model: Syntactic Orde
red Memory (SOM). The model explicitly models the structure with an incremental parser and maintains the conditional probability setting of a standard language model (left-to-right). To train the incremental parser and avoid exposure bias, we also propose a novel dynamic oracle, so that SOM is more robust to wrong parsing decisions. Experiments show that SOM can achieve strong results in language modeling, incremental parsing, and syntactic generalization tests while using fewer parameters than other models.
Contextualized representations based on neural language models have furthered the state of the art in various NLP tasks. Despite its great success, the nature of such representations remains a mystery. In this paper, we present an empirical property
of these representations---''average'' approximates first principal component''. Specifically, experiments show that the average of these representations shares almost the same direction as the first principal component of the matrix whose columns are these representations. We believe this explains why the average representation is always a simple yet strong baseline. Our further examinations show that this property also holds in more challenging scenarios, for example, when the representations are from a model right after its random initialization. Therefore, we conjecture that this property is intrinsic to the distribution of representations and not necessarily related to the input structure. We realize that these representations empirically follow a normal distribution for each dimension, and by assuming this is true, we demonstrate that the empirical property can be in fact derived mathematically.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
