Few-Shot Table-to-Text Generation with Prototype Memory

Neural table-to-text generation models have achieved remarkable progress on an array of tasks. However, due to the data-hungry nature of neural models, their performances strongly rely on large-scale training examples, limiting their applicability in real-world applications. To address this, we propose a new framework: Prototype-to-Generate (P2G), for table-to-text generation under the few-shot scenario. The proposed framework utilizes the retrieved prototypes, which are jointly selected by an IR system and a novel prototype selector to help the model bridging the structural gap between tables and texts. Experimental results on three benchmark datasets with three state-of-the-art models demonstrate that the proposed framework significantly improves the model performance across various evaluation metrics.

Memory-efficient Transformers via Top-k Attention

Following the success of dot-product attention in Transformers, numerous approximations have been recently proposed to address its quadratic complexity with respect to the input length. While these variants are memory and compute efficient, it is not possible to directly use them with popular pre-trained language models trained using vanilla attention, without an expensive corrective pre-training stage. In this work, we propose a simple yet highly accurate approximation for vanilla attention. We process the queries in chunks, and for each query, compute the top-*k* scores with respect to the keys. Our approach offers several advantages: (a) its memory usage is linear in the input size, similar to linear attention variants, such as Performer and RFA (b) it is a drop-in replacement for vanilla attention that does not require any corrective pre-training, and (c) it can also lead to significant memory savings in the feed-forward layers after casting them into the familiar query-key-value framework. We evaluate the quality of top-*k* approximation for multi-head attention layers on the Long Range Arena Benchmark, and for feed-forward layers of T5 and UnifiedQA on multiple QA datasets. We show our approach leads to accuracy that is nearly-identical to vanilla attention in multiple setups including training from scratch, fine-tuning, and zero-shot inference.

Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup

Contrastive learning has been applied successfully to learn vector representations of text. Previous research demonstrated that learning high-quality representations benefits from batch-wise contrastive loss with a large number of negatives. In practice, the technique of in-batch negative is used, where for each example in a batch, other batch examples' positives will be taken as its negatives, avoiding encoding extra negatives. This, however, still conditions each example's loss on all batch examples and requires fitting the entire large batch into GPU memory. This paper introduces a gradient caching technique that decouples backpropagation between contrastive loss and the encoder, removing encoder backward pass data dependency along the batch dimension. As a result, gradients can be computed for one subset of the batch at a time, leading to almost constant memory usage.

A Comparison of the Word Similarity Measurement in English-Arabic Translation Memory Segment Retrieval Including an Inflectional Affix Intervention

The aim of this paper is to investigate the similarity measurement approach of translation memory (TM) in five representative computer-aided translation (CAT) tools when retrieving inflectional verb-variation sentences in Arabic to English translation. In English, inflectional affixes in verbs include suffixes only; unlike English, verbs in Arabic derive voice, mood, tense, number and person through various inflectional affixes e.g. pre or post a verb root. The research question focuses on establishing whether the TM similarity algorithm measures a combination of the inflectional affixes as a word or as a character intervention when retrieving a segment. If it is dealt with as a character intervention, are the types of intervention penalized equally or differently? This paper experimentally examines, through a black box testing methodology and a test suite instrument, the penalties that TM systems' current algorithms impose when input segments and retrieved TM sources are exactly the same, except for a difference in an inflectional affix. It would be expected that, if TM systems had some linguistic knowledge, the penalty would be very light, which would be useful to translators, since a high-scoring match would be presented near the top of the list of proposals. However, analysis of TM systems' output shows that inflectional affixes are penalized more heavily than expected, and in different ways. They may be treated as an intervention on the whole word, or as a single character change.

Explicitly Modeling Syntax in Language Models with Incremental Parsing and a Dynamic Oracle

Syntax is fundamental to our thinking about language. Failing to capture the structure of input language could lead to generalization problems and over-parametrization. In the present work, we propose a new syntax-aware language model: Syntactic Ordered Memory (SOM). The model explicitly models the structure with an incremental parser and maintains the conditional probability setting of a standard language model (left-to-right). To train the incremental parser and avoid exposure bias, we also propose a novel dynamic oracle, so that SOM is more robust to wrong parsing decisions. Experiments show that SOM can achieve strong results in language modeling, incremental parsing, and syntactic generalization tests while using fewer parameters than other models.

Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation

Policy gradient algorithms have found wide adoption in NLP, but have recently become subject to criticism, doubting their suitability for NMT. Choshen et al. (2020) identify multiple weaknesses and suspect that their success is determined by the shape of output distributions rather than the reward. In this paper, we revisit these claims and study them under a wider range of configurations. Our experiments on in-domain and cross-domain adaptation reveal the importance of exploration and reward scaling, and provide empirical counter-evidence to these claims.

Memory-Based Semantic Parsing

Abstract We present a memory-based model for context- dependent semantic parsing. Previous approaches focus on enabling the decoder to copy or modify the parse from the previous utterance, assuming there is a dependency between the current and previous parses. In this work, we propose to represent contextual information using an external memory. We learn a context memory controller that manages the memory by maintaining the cumulative meaning of sequential user utterances. We evaluate our approach on three semantic parsing benchmarks. Experimental results show that our model can better process context-dependent information and demonstrates improved performance without using task-specific decoders.

Augmenting Transformers with KNN-Based Composite Memory for Dialog

Various machine learning tasks can benefit from access to external information of different modalities, such as text and images. Recent work has focused on learning architectures with large memories capable of storing this knowledge. We propose augmenting generative Transformer neural networks with KNN-based Information Fetching (KIF) modules. Each KIF module learns a read operation to access fixed external knowledge. We apply these modules to generative dialog modeling, a challenging task where information must be flexibly retrieved and incorporated to maintain the topic and flow of conversation. We demonstrate the effectiveness of our approach by identifying relevant knowledge required for knowledgeable but engaging dialog from Wikipedia, images, and human-written dialog utterances, and show that leveraging this retrieved information improves model performance, measured by automatic and human evaluation.

The Willingness to forget; Language and history in poul Ricoure

This research Tackles Ricour's perspective of the willingness to forget and the ability of memoey to step ahead the idea if past representation and looking into it as it is one future project. This future which enables humans to recall and forget, forgive and tolerate .all of this is done under what's called by Ricour" the able ego". Afterwards, it reaches a happy memory where by forgetfulness is one image out of many stored in the memory. According to this, the research discusses the beginning of relation ship between history and memory as it is perceived by Ricour, one of interrelation then it discusses the relation of memory and language via its narratives. Narrative, according to Ricour, is means which people use to express their unforgettable past experience, that they need to resurrect the past via narrating and legislating it in stories. Then, it proceeds to handle the idea of willingness to forget and the relation which links forgetfulness to memory, which is one holistic relation, according to Ricour. This means that there is no forgetfulness without recalling and vice versa. Despit that, as perceived by Ricour, memory has always been living in confusion/ hesitation. In other word, you hesitate as to what to recall and what to forget. This has forced or led this research of memory as it is one future project as Ricour assures.it finally concludes with some findings that we sought to have in regards to Ricour's perspective regarding this issus.

الجولان و الذاكرة الوطنية

الجولان ليس مجرد هضبة و جبال و سهول و وديان تبلغ مساحتها ( 1860 كم 2 و قد وقع تحت الاحتلال الصهيوني منها نحو ( 1254 كم 2)، ثم انسحبت قواته من مساحة ( 100 كم 2) بموجب اتفاقية الفصل عام ( 1974 م)... و لا هو مجرد مياه و أشجار، تزين طبيعتها و أضرحة و آثار؛ و كنائس و مساجد و عمران تغني تاريخها.. و ليست مجرد قطعة من الأرض تمتاز بموقعها الجغرافي؛ و سجلها التاريخي الثري بالأحداث و الانتصارات التي انعقدت على رايات الجيوش و لا سيما العربية الإسلامية...