Generic Mechanism for Reducing Repetitions in Encoder-Decoder Models

Encoder-decoder models have been commonly used for many tasks such as machine translation and response generation. As previous research reported, these models suffer from generating redundant repetition. In this research, we propose a new mechanism for encoder-decoder models that estimates the semantic difference of a source sentence before and after being fed into the encoder-decoder model to capture the consistency between two sides. This mechanism helps reduce repeatedly generated tokens for a variety of tasks. Evaluation results on publicly available machine translation and response generation datasets demonstrate the effectiveness of our proposal.

On Reducing Repetition in Abstractive Summarization

Repetition in natural language generation reduces the informativeness of text and makes it less appealing. Various techniques have been proposed to alleviate it. In this work, we explore and propose techniques to reduce repetition in abstractive summarization. First, we explore the application of unlikelihood training and embedding matrix regularizers from previous work on language modeling to abstractive summarization. Next, we extend the coverage and temporal attention mechanisms to the token level to reduce repetition. In our experiments on the CNN/Daily Mail dataset, we observe that these techniques reduce the amount of repetition and increase the informativeness of the summaries, which we confirm via human evaluation.

On Attention Redundancy: A Comprehensive Study

Multi-layer multi-head self-attention mechanism is widely applied in modern neural language models. Attention redundancy has been observed among attention heads but has not been deeply studied in the literature. Using BERT-base model as an example, this paper provides a comprehensive study on attention redundancy which is helpful for model interpretation and model compression. We analyze the attention redundancy with Five-Ws and How. (What) We define and focus the study on redundancy matrices generated from pre-trained and fine-tuned BERT-base model for GLUE datasets. (How) We use both token-based and sentence-based distance functions to measure the redundancy. (Where) Clear and similar redundancy patterns (cluster structure) are observed among attention heads. (When) Redundancy patterns are similar in both pre-training and fine-tuning phases. (Who) We discover that redundancy patterns are task-agnostic. Similar redundancy patterns even exist for randomly generated token sequences. (Why'') We also evaluate influences of the pre-training dropout ratios on attention redundancy. Based on the phase-independent and task-agnostic attention redundancy patterns, we propose a simple zero-shot pruning method as a case study. Experiments on fine-tuning GLUE tasks verify its effectiveness. The comprehensive analyses on attention redundancy make model understanding and zero-shot model pruning promising.

Structure of repetition in suspended ى Zuhair bin Abi Salma

This study aims to study the phenomenon of repetition in the commentator Zuhair ibn Abi Salma, a phenomenon that is clearly manifested in his comment, which is related to some extent closely to the structure of the poet psychological and existential, as the repetition of the poet chooses a combination of linguistic structures and stylistic elements. A selective process of language-language spirit, which reveals the secret of his tendency to this stylistic style.

التّكرار في شعر حُمَيد بِن ثَور الهِلاليّ

this study aim to discover the formatiey of sound and to analyize semantics to find out the whole and partial image .

Combining between rhetoric and poetry at the Gahleon and its impact on poetry

This research deals with the phenomenon of combining rhetoric and poetry at a number of the Gahleon, and the statement of the effect of this combining in their poetry, explaining through the issue of conflict of prestige betweenthe orator and the poet in pre-Islamic era, and how this conflict has been compromised between who combined the technocracy, as the research shows the reason oflack of those who gathered between technocracy. The effect of combining in poetry is dealt with research on two levels: the stylistic level, and shows all forms of repetition: repeating letters, words and methods. While appears in the substantive level (moral): the pursuit of incomprehensible, the concentration on thinking and subjects, and the large number of wisdom and wills.

جماليات التّكرار في القصّة القرآنيّة

تناول هذا البحث جماليات تكرار الأحداث في قصص القرآن، فبدأ بعرض مفهوم التٌكرار في اللغة، ثم أوضح معنى التٌكرار في القصٌة أو الرواية، و بيٌن أنواعه الثلاثة المتمثلة في التٌكرار الزائد و تكرار الحدث و تكرار السرد، ثم ذكر معنى التواتر أو التردد، و الاحتمالات القائمة عليه .

Study the Effect of Rainfall Characteristics (raindrop effect – rainfall intensity - rainfall relay ) on the Water Erosion of the Soil

This research was conducted in order to determine the impact of raindrops in terms of force of impact and its relation to rain intensity as well as the relay rain on the amount of soil eroded and water drifting due to water erosion .

Numerical Solution of Some Important Models of Partial Differential Equations Using Approximate - Analytical Methods ( ADM – VIM )

In this article, powerful approximate analytical methods, called Adomian decomposition method and variational iteration method are introduced and applied to obtaining the approximate analytical solutions for an important models of linear and non-linear partial differential equations such as ( nonlinear Klein Gordon equation - nonlinear wave equation - linear telegraph equation - nonlinear diffusion convection equation ) . The studied examples are used to reveal that those methods are very effective and convenient for solving linear and nonlinear partial differential equations . Numerical results and comparisons with the exact solution are included to show validity, ability, accuracy, strength and effectiveness of those techniques.

The Development of Precipitation Relationships of Different Recurrence Intervals in Aleppo Governorate

The aim of this study is to determine the best probability distribution of annual, monthly, annual one day maximum precipitation for stations in Aleppo Governorate by using nw2 test, then estimating annual, monthly, one day maximum precipitation to various return periods according to the best probability distribution, and using Chow’s general frequency formula.