Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userLearning Sparse Sentence Encoding without Supervision: An Exploration of Sparsity in Variational Autoencoders
التعلم جملة متلقجة تشفير دون إشراف: استكشاف Sparsity في السيارات الآلية المتنوعة
293
0 0
0.0
(
0
)
Created by
Shamra Editor
Ask ChatGPT about the research
لقد كانت معروفة منذ فترة طويلة أن Sparsity هي تحيز حثي فعال لتعلم التمثيل الفعال للبيانات في المتجهات ذات الأبعاد الثابتة، وقد تم استكشافها في العديد من مجالات التعلم التمثيل. من اهتمام خاص بهذا العمل هو التحقيق في Sparsity ضمن إطار VAE الذي تم استكشافه كثيرا في مجال الصورة، ولكنه كان يفتقر إلى مستوى الاستكشاف الأساسي في NLP. بالإضافة إلى ذلك، يتخلف NLP أيضا من حيث تعلم تمثيلات متفرق لوحدات نصية كبيرة على سبيل المثال، الجمل. نحن نستخدم VAES التي تحفز التمثيلات الكامنة المتفرقة لوحدات نصية كبيرة لمعالجة أوجه القصور المذكورة أعلاه. أولا، ننتقل في هذا الاتجاه من خلال قياس نجاح الحالة غير المعردة للدولة (SOTA) وغيرها من خطوط الأساس السريع في VAE للنص واقتراح نموذج VIE هرمي متفرق لمعالجة مشكلة الاستقرار في سوتا. بعد ذلك، ننظر إلى آثار Sparsity على تصنيف النص عبر 3 مجموعات من مجموعات البيانات، وتسليط الضوء على ارتباط بين أداء التمثيلات الكامنة المتفرعة حول مهام المصب وقدرته على تشفير المعلومات المتعلقة بالمهام.
It has been long known that sparsity is an effective inductive bias for learning efficient representation of data in vectors with fixed dimensionality, and it has been explored in many areas of representation learning. Of particular interest to this work is the investigation of the sparsity within the VAE framework which has been explored a lot in the image domain, but has been lacking even a basic level of exploration in NLP. Additionally, NLP is also lagging behind in terms of learning sparse representations of large units of text e.g., sentences. We use the VAEs that induce sparse latent representations of large units of text to address the aforementioned shortcomings. First, we move in this direction by measuring the success of unsupervised state-of-the-art (SOTA) and other strong VAE-based sparsification baselines for text and propose a hierarchical sparse VAE model to address the stability issue of SOTA. Then, we look at the implications of sparsity on text classification across 3 datasets, and highlight a link between performance of sparse latent representations on downstream tasks and its ability to encode task-related information.
References used
https://aclanthology.org/
rate research
Read More
Variational autoencoders have been studied as a promising approach to model one-to-many mappings from context to response in chat response generation. However, they often fail to learn proper mappings. One of the reasons for this failure is the discr
epancy between a response and a latent variable sampled from an approximated distribution in training. Inappropriately sampled latent variables hinder models from constructing a modulated latent space. As a result, the models stop handling uncertainty in conversations. To resolve that, we propose speculative sampling of latent variables. Our method chooses the most probable one from redundantly sampled latent variables for tying up the variable with a given response. We confirm the efficacy of our method in response generation with massive dialogue data constructed from Twitter posts.
Text variational autoencoders (VAEs) are notorious for posterior collapse, a phenomenon where the model's decoder learns to ignore signals from the encoder. Because posterior collapse is known to be exacerbated by expressive decoders, Transformers ha
ve seen limited adoption as components of text VAEs. Existing studies that incorporate Transformers into text VAEs (Li et al., 2020; Fang et al., 2021) mitigate posterior collapse using massive pretraining, a technique unavailable to most of the research community without extensive computing resources. We present a simple two-phase training scheme to convert a sequence-to-sequence Transformer into a VAE with just finetuning. The resulting language model is competitive with massively pretrained Transformer-based VAEs in some internal metrics while falling short on others. To facilitate training we comprehensively explore the impact of common posterior collapse alleviation techniques in the literature. We release our code for reproducability.
The ability of learning disentangled representations represents a major step for interpretable NLP systems as it allows latent linguistic features to be controlled. Most approaches to disentanglement rely on continuous variables, both for images and
text. We argue that despite being suitable for image datasets, continuous variables may not be ideal to model features of textual data, due to the fact that most generative factors in text are discrete. We propose a Variational Autoencoder based method which models language features as discrete variables and encourages independence between variables for learning disentangled representations. The proposed model outperforms continuous and discrete baselines on several qualitative and quantitative benchmarks for disentanglement as well as on a text style transfer downstream application.
Many NLP models operate over sequences of subword tokens produced by hand-crafted tokenization rules and heuristic subword induction algorithms. A simple universal alternative is to represent every computerized text as a sequence of bytes via UTF-8,
obviating the need for an embedding layer since there are fewer token types (256) than dimensions. Surprisingly, replacing the ubiquitous embedding layer with one-hot representations of each byte does not hurt performance; experiments on byte-to-byte machine translation from English to 10 different languages show a consistent improvement in BLEU, rivaling character-level and even standard subword-level models. A deeper investigation reveals that the combination of embeddingless models with decoder-input dropout amounts to token dropout, which benefits byte-to-byte models in particular.
In this paper we study pejorative language, an under-explored topic in computational linguistics. Unlike existing models of offensive language and hate speech, pejorative language manifests itself primarily at the lexical level, and describes a word
that is used with a negative connotation, making it different from offensive language or other more studied categories. Pejorativity is also context-dependent: the same word can be used with or without pejorative connotations, thus pejorativity detection is essentially a problem similar to word sense disambiguation. We leverage online dictionaries to build a multilingual lexicon of pejorative terms for English, Spanish, Italian, and Romanian. We additionally release a dataset of tweets annotated for pejorative use. Based on these resources, we present an analysis of the usage and occurrence of pejorative words in social media, and present an attempt to automatically disambiguate pejorative usage in our dataset.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
