Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userCan the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?
هل يمكن استخدام المحولات كحل قطرة لاستخراج RNNS في Ganes التي توليد النص؟
327
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
في هذه الورقة، نتعلم مشكلة توليد النصوص الدقيقة بميزانية حسابية محدودة.لذلك، نستخدم هندسة شبكة مصممة ذات أداء أداء جيدا (GAN) - Gan التي ترويج التنوع (DPGAN)، وحاول استبدال قطرة LSTM بطبقة محول ذاتية انتباهي من أجل الرافعة الماليةكفاءتهم.تم تقييم DPGan الناتج عن النفس (SADPGAN) للأداء والجودة والتنوع للنص والاستقرار الناتج.تشير التجارب الحاسوبية إلى أن بنية محول غير قادرة على الاسترجاع في استبدال طبقة LSTM، ضمان الأداء أثناء مرحلة التدريب المسبق وتخضع لانهيار الوضع الكامل أثناء مرحلة ضبط GAN.تشير نتائجنا إلى أن الهندسة المعمارية المحول تحتاج إلى تكييفها قبل أن يتم استخدامها كإعداد لاستبدال RNNS في قانع إنشاء النصوص.
In this paper we address the problem of fine-tuned text generation with a limited computational budget. For that, we use a well-performing text generative adversarial network (GAN) architecture - Diversity-Promoting GAN (DPGAN), and attempted a drop-in replacement of the LSTM layer with a self-attention-based Transformer layer in order to leverage their efficiency. The resulting Self-Attention DPGAN (SADPGAN) was evaluated for performance, quality and diversity of generated text and stability. Computational experiments suggested that a transformer architecture is unable to drop-in replace the LSTM layer, under-performing during the pre-training phase and undergoing a complete mode collapse during the GAN tuning phase. Our results suggest that the transformer architecture need to be adapted before it can be used as a replacement for RNNs in text-generating GANs.
References used
https://aclanthology.org/
rate research
Read More
Vector representations have become a central element in semantic language modelling, leading to mathematical overlaps with many fields including quantum theory. Compositionality is a core goal for such representations: given representations for wet'
and fish', how should the concept wet fish' be represented? This position paper surveys this question from two points of view. The first considers the question of whether an explicit mathematical representation can be successful using only tools from within linear algebra, or whether other mathematical tools are needed. The second considers whether semantic vector composition should be explicitly described mathematically, or whether it can be a model-internal side-effect of training a neural network. A third and newer question is whether a compositional model can be implemented on a quantum computer. Given the fundamentally linear nature of quantum mechanics, we propose that these questions are related, and that this survey may help to highlight candidate operations for future quantum implementation.
This muscle has
been used to design a robot arm similar to that of the humerus and
forearm, which can attract large objects with a weight of up to 500
N, which is equivalent to a professional bodybuilder raising his
weight.
The research aims to optimize the investment in solar cooling process using two
models of vessels (clay- mineral).The study was conducted at the site of Tartous in the
month (the fourth - fifth - sixth) years (2013) and that the fruits of the tomat
o study, she
stated that the pottery is causing a drop in temperature between )4-6( degrees Celsius, and
that the metal causes the low temperature range between (3-5 ) degrees Celsius although
the fruits of tomatoes preserved pottery vessels have not undergone any damage of its
structure or texture during the period of conservation (27 days) compared to the control
which is exposed to damage during the (12 days) .
In this paper, we focus on the detection of sexist hate speech against women in tweets studying for the first time the impact of gender stereotype detection on sexism classification. We propose: (1) the first dataset annotated for gender stereotype d
etection, (2) a new method for data augmentation based on sentence similarity with multilingual external datasets, and (3) a set of deep learning experiments first to detect gender stereotypes and then, to use this auxiliary task for sexism detection. Although the presence of stereotypes does not necessarily entail hateful content, our results show that sexism classification can definitively benefit from gender stereotype detection.
Despite the recent advances in applying pre-trained language models to generate high-quality texts, generating long passages that maintain long-range coherence is yet challenging for these models. In this paper, we propose DiscoDVT, a discourse-aware
discrete variational Transformer to tackle the incoherence issue. DiscoDVT learns a discrete variable sequence that summarizes the global structure of the text and then applies it to guide the generation process at each decoding step. To further embed discourse-aware information into the discrete latent representations, we introduce an auxiliary objective to model the discourse relations within the text. We conduct extensive experiments on two open story generation datasets and demonstrate that the latent codes learn meaningful correspondence to the discourse structures that guide the model to generate long texts with better long-range coherence.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
