Finetuning Pretrained Transformers into Variational Autoencoders

Text variational autoencoders (VAEs) are notorious for posterior collapse, a phenomenon where the model's decoder learns to ignore signals from the encoder. Because posterior collapse is known to be exacerbated by expressive decoders, Transformers have seen limited adoption as components of text VAEs. Existing studies that incorporate Transformers into text VAEs (Li et al., 2020; Fang et al., 2021) mitigate posterior collapse using massive pretraining, a technique unavailable to most of the research community without extensive computing resources. We present a simple two-phase training scheme to convert a sequence-to-sequence Transformer into a VAE with just finetuning. The resulting language model is competitive with massively pretrained Transformer-based VAEs in some internal metrics while falling short on others. To facilitate training we comprehensively explore the impact of common posterior collapse alleviation techniques in the literature. We release our code for reproducability.

Learning Sparse Sentence Encoding without Supervision: An Exploration of Sparsity in Variational Autoencoders

It has been long known that sparsity is an effective inductive bias for learning efficient representation of data in vectors with fixed dimensionality, and it has been explored in many areas of representation learning. Of particular interest to this work is the investigation of the sparsity within the VAE framework which has been explored a lot in the image domain, but has been lacking even a basic level of exploration in NLP. Additionally, NLP is also lagging behind in terms of learning sparse representations of large units of text e.g., sentences. We use the VAEs that induce sparse latent representations of large units of text to address the aforementioned shortcomings. First, we move in this direction by measuring the success of unsupervised state-of-the-art (SOTA) and other strong VAE-based sparsification baselines for text and propose a hierarchical sparse VAE model to address the stability issue of SOTA. Then, we look at the implications of sparsity on text classification across 3 datasets, and highlight a link between performance of sparse latent representations on downstream tasks and its ability to encode task-related information.

Extractive Opinion Summarization in Quantized Transformer Spaces

Abstract We present the Quantized Transformer (QT), an unsupervised system for extractive opinion summarization. QT is inspired by Vector- Quantized Variational Autoencoders, which we repurpose for popularity-driven summarization. It uses a clustering interpretation of the quantized space and a novel extraction algorithm to discover popular opinions among hundreds of reviews, a significant step towards opinion summarization of practical scope. In addition, QT enables controllable summarization without further training, by utilizing properties of the quantized space to extract aspect-specific summaries. We also make publicly available Space, a large-scale evaluation benchmark for opinion summarizers, comprising general and aspect-specific summaries for 50 hotels. Experiments demonstrate the promise of our approach, which is validated by human studies where judges showed clear preference for our method over competitive baselines.

Traffic Density Prediction state of the art

ازدادت الحاجة لأنظمة التنبؤ المرورية وأصبحت حاجة ضرورية وملحة في أنظمة إدارة المرور المتقدمة، ذلك لأن توقع كثافة المرور يقلل الازدحام المروري ويسهل حركة السير. ومع وجود تنبؤ دقيق بحالة المرور سيكون بمقدورنا تطوير نظام إدارة مرورية متطور ونظام استعلامات متطور للمسافرين. التحدي الذي يواجه مشكلة نمذجة حالة المرور هو الخصائص المعقدة للعمليات المرورية العشوائية. معلومات التسلسل الزمني للكثافة المرورية، والسرعات، والتمركز المروري والتي يتم جمعها من مواقع مختلفة تمتلك خصائص مختلفة عن بعضها، وبذلك عملية التنبؤ بالكثافة المرورية المستقبلية ليست عملية بديهية، ويناقش هذا البحث عدة طرق قامت بتقديم حلول لهذه المشكلة.