Adversarial Robustness of Deep Learning: Theory, Algorithms, and Applications

This tutorial aims to introduce the fundamentals of adversarial robustness of deep learning, presenting a well-structured review of up-to-date techniques to assess the vulnerability of various types of deep learning models to adversarial examples. This tutorial will particularly highlight state-of-the-art techniques in adversarial attacks and robustness verification of deep neural networks (DNNs). We will also introduce some effective countermeasures to improve the robustness of deep learning models, with a particular focus on adversarial training. We aim to provide a comprehensive overall picture about this emerging direction and enable the community to be aware of the urgency and importance of designing robust deep learning models in safety-critical data analytical applications, ultimately enabling the end-users to trust deep learning classifiers. We will also summarize potential research directions concerning the adversarial robustness of deep learning, and its potential benefits to enable accountable and trustworthy deep learning-based data analytical systems and applications.

Topological Pilot Assignment in Large-Scale Distributed MIMO Networks

We consider the pilot assignment problem in large-scale distributed multi-input multi-output (MIMO) networks, where a large number of remote radio head (RRH) antennas are randomly distributed in a wide area, and jointly serve a relatively smaller number of users (UE) coherently. By artificially imposing topological structures on the UE-RRH connectivity, we model the network by a partially-connected interference network, so that the pilot assignment problem can be cast as a topological interference management problem with multiple groupcast messages. Building upon such connection, we formulate the topological pilot assignment (TPA) problem in two different ways with respect to whether or not the to-be-estimated channel connectivity pattern is known a priori. When it is known, we formulate the TPA problem as a low-rank matrix completion problem that can be solved by a simple alternating projection algorithm. Otherwise, we formulate it as a sequential maximum weight induced matching problem that can be solved by either a mixed integer linear program or a simple yet efficient greedy algorithm. With respect to two different formulations of the TPA problem, we evaluate the efficiency of the proposed algorithms under the cell-free massive MIMO setting.

Downlink Precoding for DP-UPA FDD Massive MIMO via Multi-Dimensional Active Channel Sparsification

In this paper, we consider user selection and downlink precoding for an over-loaded single-cell massive multiple-input multiple-output (MIMO) system in frequency division duplexing (FDD) mode, where the base station is equipped with a dual-polarized uniform planar array (DP-UPA) and serves a large number of single-antenna users. Due to the absence of uplink-downlink channel reciprocity and the high-dimensionality of channel matrices, it is extremely challenging to design downlink precoders using closed-loop channel probing and feedback with limited spectrum resource. To address these issues, a novel methodology -- active channel sparsification (ACS) -- has been proposed recently in the literature for uniform linear array (ULA) to design sparsifying precoders, which boosts spectral efficiency for multi-user downlink transmission with substantially reduced channel feedback overhead. Pushing forward this line of research, we aim to facilitate the potential deployment of ACS in practical FDD massive MIMO systems, by extending it from ULA to DP-UPA with explicit user selection and making the current ACS implementation simplified. To this end, by leveraging Toeplitz structure of channel covariance matrices, we extend the original ACS using scale-weight bipartite graph representation to the matrix-weight counterpart. Building upon this, we propose a multi-dimensional ACS (MD-ACS) method, which is a generalization of original ACS formulation and is more suitable for DP-UPA antenna configurations. The nonlinear integer program formulation of MD-ACS can be classified as a generalized multi-assignment problem (GMAP), for which we propose a simple yet efficient greedy algorithm to solve it. Simulation results demonstrate the performance improvement of the proposed MD-ACS with greedy algorithm over the state-of-the-art methods based on the QuaDRiGa channel models.

LGD-GCN: Local and Global Disentangled Graph Convolutional Networks

Disentangled Graph Convolutional Network (DisenGCN) is an encouraging framework to disentangle the latent factors arising in a real-world graph. However, it relies on disentangling information heavily from a local range (i.e., a node and its 1-hop neighbors), while the local information in many cases can be uneven and incomplete, hindering the interpretabiliy power and model performance of DisenGCN. In this paper, we introduce a novel Local and Global Disentangled Graph Convolutional Network (LGD-GCN) to capture both local and global information for graph disentanglement. LGD-GCN performs a statistical mixture modeling to derive a factor-aware latent continuous space, and then constructs different structures w.r.t. different factors from the revealed space. In this way, the global factor-specific information can be efficiently and selectively encoded via a message passing along these built structures, strengthening the intra-factor consistency. We also propose a novel diversity promoting regularizer employed with the latent space modeling, to encourage inter-factor diversity. Evaluations of the proposed LGD-GCN on the synthetic and real-world datasets show a better interpretability and improved performance in node classification over the existing competitive models.

A Little Energy Goes a Long Way: Energy-Efficient, Accurate Conversion from Convolutional Neural Networks to Spiking Neural Networks

Spiking neural networks (SNNs) offer an inherent ability to process spatial-temporal data, or in other words, realworld sensory data, but suffer from the difficulty of training high accuracy models. A major thread of research on SNNs is on converting a pre-trained convolutional neural network (CNN) to an SNN of the same structure. State-of-the-art conversion methods are approaching the accuracy limit, i.e., the near-zero accuracy loss of SNN against the original CNN. However, we note that this is made possible only when significantly more energy is consumed to process an input. In this paper, we argue that this trend of energy for accuracy is not necessary -- a little energy can go a long way to achieve the near-zero accuracy loss. Specifically, we propose a novel CNN-to-SNN conversion method that is able to use a reasonably short spike train (e.g., 256 timesteps for CIFAR10 images) to achieve the near-zero accuracy loss. The new conversion method, named as explicit current control (ECC), contains three techniques (current normalisation, thresholding for residual elimination, and consistency maintenance for batch-normalisation), in order to explicitly control the currents flowing through the SNN when processing inputs. We implement ECC into a tool nicknamed SpKeras, which can conveniently import Keras CNN models and convert them into SNNs. We conduct an extensive set of experiments with the tool -- working with VGG16 and various datasets such as CIFAR10 and CIFAR100 -- and compare with state-of-the-art conversion methods. Results show that ECC is a promising method that can optimise over energy consumption and accuracy loss simultaneously.

Topological Interference Management with Adversarial Perturbation

In this paper, we consider the topological interference management (TIM) problem in a dynamic setting, where an adversary perturbs network topology to prevent the exploitation of sophisticated coding opportunities (e.g., interference alignment). Focusing on a special class of network topology - chordal networks - we investigate algorithmic aspects of the TIM problem under adversarial topology perturbation. In particular, given the adversarial perturbation with respect to edge insertion/deletion, we propose a dynamic graph coloring algorithm that allows for a constant number of re-coloring updates against each inserted/deleted edge to achieve the information-theoretic optimality. This is a sharp reduction of the general graph re-coloring, whose optimal number of updates scales as the size of the network, thanks to the delicate exploitation of the structural properties of weakly chordal graphs.

How does Weight Correlation Affect the Generalisation Ability of Deep Neural Networks

This paper studies the novel concept of weight correlation in deep neural networks and discusses its impact on the networks generalisation ability. For fully-connected layers, the weight correlation is defined as the average cosine similarity between weight vectors of neurons, and for convolutional layers, the weight correlation is defined as the cosine similarity between filter matrices. Theoretically, we show that, weight correlation can, and should, be incorporated into the PAC Bayesian framework for the generalisation of neural networks, and the resulting generalisation bound is monotonic with respect to the weight correlation. We formulate a new complexity measure, which lifts the PAC Bayes measure with weight correlation, and experimentally confirm that it is able to rank the generalisation errors of a set of networks more precisely than existing measures. More importantly, we develop a new regulariser for training, and provide extensive experiments that show that the generalisation error can be greatly reduced with our novel approach.

Asymptotic Singular Value Distribution of Linear Convolutional Layers

In convolutional neural networks, the linear transformation of multi-channel two-dimensional convolutional layers with linear convolution is a block matrix with doubly Toeplitz blocks. Although a wrapping around operation can transform linear convolution to a circular one, by which the singular values can be approximated with reduced computational complexity by those of a block matrix with doubly circulant blocks, the accuracy of such an approximation is not guaranteed. In this paper, we propose to inspect such a linear transformation matrix through its asymptotic spectral representation - the spectral density matrix - by which we develop a simple singular value approximation method with improved accuracy over the circular approximation, as well as upper bounds for spectral norm with reduced computational complexity. Compared with the circular approximation, we obtain moderate improvement with a subtle adjustment of the singular value distribution. We also demonstrate that the spectral norm upper bounds are effective spectral regularizers for improving generalization performance in ResNets.

Robust Precoding in Massive MIMO: A Deep Learning Approach

In this paper, we consider massive multiple-input-multiple-output (MIMO) communication systems with a uniform planar array (UPA) at the base station (BS) and investigate the downlink precoding with imperfect channel state information (CSI). By exploiting both instantaneous and statistical CSI, we aim to design precoding vectors to maximize the ergodic rate (e.g., sum rate, minimum rate and etc.) subject to a total transmit power constraint. To maximize an upper bound of the ergodic rate, we leverage the corresponding Lagrangian formulation and identify the structural characteristics of the optimal precoder as the solution to a generalized eigenvalue problem. As such, the high-dimensional precoder design problem turns into a low-dimensional power control problem. The Lagrange multipliers play a crucial role in determining both precoder directions and power parameters, yet are challenging to be solved directly. To figure out the Lagrange multipliers, we develop a general framework underpinned by a properly designed neural network that learns directly from CSI. To further relieve the computational burden, we obtain a low-complexity framework by decomposing the original problem into computationally efficient subproblems with instantaneous and statistical CSI handled separately. With the off-line pretrained neural network, the online computational complexity of precoding is substantially reduced compared with the existing iterative algorithm while maintaining nearly the same performance.

FDD Massive MIMO via UL/DL Channel Covariance Extrapolation and Active Channel Sparsification

We propose a novel method for massive Multiple-Input Multiple-Output (massive MIMO) in Frequency Division Duplexing (FDD) systems. Due to the large frequency separation between Uplink (UL) and Downlink (DL), in FDD systems channel reciprocity does not hold. Hence, in order to provide DL channel state information to the Base Station (BS), closed-loop DL channel probing and Channel State Information (CSI) feedback is needed. In massive MIMO this incurs typically a large training overhead. For example, in a typical configuration with M = 200 BS antennas and fading coherence block of T = 200 symbols, the resulting rate penalty factor due to the DL training overhead, given by max{0, 1 - M/T}, is close to 0. To reduce this overhead, we build upon the well-known fact that the Angular Scattering Function (ASF) of the user channels is invariant over frequency intervals whose size is small with respect to the carrier frequency (as in current FDD cellular standards). This allows to estimate the users DL channel covariance matrix from UL pilots without additional overhead. Based on this covariance information, we propose a novel sparsifying precoder in order to maximize the rank of the effective sparsified channel matrix subject to the condition that each effective user channel has sparsity not larger than some desired DL pilot dimension T_{dl}, resulting in the DL training overhead factor max{0, 1 - T_{dl} / T} and CSI feedback cost of T_{dl} pilot measurements. The optimization of the sparsifying precoder is formulated as a Mixed Integer Linear Program, that can be efficiently solved. Extensive simulation results demonstrate the superiority of the proposed approach with respect to concurrent state-of-the-art schemes based on compressed sensing or UL/DL dictionary learning.