Layer-wise Model Pruning based on Mutual Information

Inspired by mutual information (MI) based feature selection in SVMs and logistic regression, in this paper, we propose MI-based layer-wise pruning: for each layer of a multi-layer neural network, neurons with higher values of MI with respect to preserved neurons in the upper layer are preserved. Starting from the top softmax layer, layer-wise pruning proceeds in a top-down fashion until reaching the bottom word embedding layer. The proposed pruning strategy offers merits over weight-based pruning techniques: (1) it avoids irregular memory access since representations and matrices can be squeezed into their smaller but dense counterparts, leading to greater speedup; (2) in a manner of top-down pruning, the proposed method operates from a more global perspective based on training signals in the top layer, and prunes each layer by propagating the effect of global signals through layers, leading to better performances at the same sparsity level. Extensive experiments show that at the same sparsity level, the proposed strategy offers both greater speedup and higher performances than weight-based pruning methods (e.g., magnitude pruning, movement pruning).

RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation

Knowledge Distillation (KD) is extensively used in Natural Language Processing to compress the pre-training and task-specific fine-tuning phases of large neural language models. A student model is trained to minimize a convex combination of the prediction loss over the labels and another over the teacher output. However, most existing works either fix the interpolating weight between the two losses apriori or vary the weight using heuristics. In this work, we propose a novel sample-wise loss weighting method, RW-KD. A meta-learner, simultaneously trained with the student, adaptively re-weights the two losses for each sample. We demonstrate, on 7 datasets of the GLUE benchmark, that RW-KD outperforms other loss re-weighting methods for KD.

Monotonic Simultaneous Translation with Chunk-wise Reordering and Refinement

Recent work in simultaneous machine translation is often trained with conventional full sentence translation corpora, leading to either excessive latency or necessity to anticipate as-yet-unarrived words, when dealing with a language pair whose word orders significantly differ. This is unlike human simultaneous interpreters who produce largely monotonic translations at the expense of the grammaticality of a sentence being translated. In this paper, we thus propose an algorithm to reorder and refine the target side of a full sentence translation corpus, so that the words/phrases between the source and target sentences are aligned largely monotonically, using word alignment and non-autoregressive neural machine translation. We then train a widely used wait-k simultaneous translation model on this reordered-and-refined corpus. The proposed approach improves BLEU scores and resulting translations exhibit enhanced monotonicity with source sentences.

Multi Output Learning using Task Wise Attention for Predicting Binary Properties of Tweets : Shared-Task-On-Fighting the COVID-19 Infodemic

In this paper, we describe our system for the shared task on Fighting the COVID-19 Infodemic in the English Language. Our proposed architecture consists of a multi-output classification model for the seven tasks, with a task-wise multi-head attention layer for inter-task information aggregation. This was built on top of the Bidirectional Encoder Representations obtained from the RoBERTa Transformer. We were able to achieve a mean F1 score of 0.891 on the test data, leading us to the second position on the test-set leaderboard.

Enhancend Secure Data Aggregation In Mobile Wireless Sensor Networks (MWSNs)

Mobile Wireless Sensor Network (MWSN) is an emerging technology for attraction of researchers with its research advantage and various application domains. Due to limited resources of sensor nodes such as transmission power, communication capability and size of memory, data aggregation algorithms are the most practical technique that reduces large amount of transmission in this network. Security is an important criterion to be considered because, wireless sensor nodes are deployed in a remote or hostile environment area that is prone to attacks easily. Therefore, security are essential issue for MWSN to protect information against attacks. In this research, we offered an algorithm of secure data aggregation in MWSN based on pair-wise keys technology and hash function. We studied important parameters such as execution time, end-to-end delay and number of storied keys. Results showed that

Reading into the Image of Laila al-Akhielieyyeh's Poetic Self

Not only poetry has been restricted to men, but also Arab woman used to taste and write poetry since time immemorial, next to her interest in prose and narratology comparable to all Arabs at that Bedouin time—when they were still leading a simple life. However, names of women have been revealed and broadly circulated concering handling and criticizing poetry since antiquity.