Remote State Estimation of Multiple Systems over Multiple Markov Fading Channels

We consider remote state estimation of multiple discrete-time linear time-invariant (LTI) systems over multiple wireless time-varying communication channels. Each system state is measured by a sensor, and the measurements from sensors are sent to a remote estimator over the shared wireless channels in a scheduled manner. We answer the following open problem: what is the fundamental requirement on the multi-sensor-multi-channel system to guarantee the existence of a sensor scheduling policy that can stabilize the remote estimation system? To tackle the problem, we propose a novel policy construction method, and develop a new analytical approach by applying the asymptotic theory of spectral radii of products of non-negative matrices. A necessary and sufficient stability condition is derived in terms of the LTI system parameters and the channel statistics, which is more effective than existing sufficient conditions available in the literature. Explicit scheduling policies with stability guarantees are presented as well. We further extend the analytical framework to cover remote estimation with four alternative network setups and obtain corresponding necessary and sufficient stability conditions.

Anytime Control under Practical Communication Model

We investigate a novel anytime control algorithm for wireless networked control with random dropouts. The controller computes sequences of tentative future control commands using time-varying (Markovian) computational resources. The sensor-controller and controller-actuator channel states are spatial- and time-correlated, and are modeled as a multi-state Markov process. To compensate for the effect of packet dropouts, a dual-buffer mechanism is proposed. We develop a novel cycle-cost-based approach to obtain the stability conditions on the nonlinear plant, controller, network and computational resources.

Remote State Estimation with Smart Sensors over Markov Fading Channels

We consider a fundamental remote state estimation problem of discrete-time linear time-invariant (LTI) systems. A smart sensor forwards its local state estimate to a remote estimator over a time-correlated $M$-state Markov fading channel, where the packet drop probability is time-varying and depends on the current fading channel state. We establish a necessary and sufficient condition for mean-square stability of the remote estimation error covariance as $rho^2(mathbf{A})rho(mathbf{DM})<1$, where $rho(cdot)$ denotes the spectral radius, $mathbf{A}$ is the state transition matrix of the LTI system, $mathbf{D}$ is a diagonal matrix containing the packet drop probabilities in different channel states, and $mathbf{M}$ is the transition probability matrix of the Markov channel states. To derive this result, we propose a novel estimation-cycle based approach, and provide new element-wise bounds of matrix powers. The stability condition is verified by numerical results, and is shown more effective than existing sufficient conditions in the literature. We observe that the stability region in terms of the packet drop probabilities in different channel states can either be convex or concave depending on the transition probability matrix $mathbf{M}$. Our numerical results suggest that the stability conditions for remote estimation may coincide for setups with a smart sensor and with a conventional one (which sends raw measurements to the remote estimator), though the smart sensor setup achieves a better estimation performance.

A Game-Theoretic Approach to Covert Communications

This paper considers a game-theoretic formulation of the covert communications problem with finite blocklength, where the transmitter (Alice) can randomly vary her transmit power in different blocks, while the warden (Willie) can randomly vary his detection threshold in different blocks. In this two player game, the payoff for Alice is a combination of the coding rate to the receiver (Bob) and the detection error probability at Willie, while the payoff for Willie is the negative of his detection error probability. Nash equilibrium solutions to the game are obtained, and shown to be efficiently computable using linear programming. For less covert requirements, our game-theoretic approach can achieve significantly higher coding rates than uniformly distributed transmit powers. We then consider the situation with an additional jammer, where Alice and the jammer can both vary their powers. We pose a two player game where Alice and the jammer jointly comprise one player, with Willie the other player. The use of a jammer is shown in numerical simulations to lead to further significant performance improvements.

An event-triggered transmission scheduling strategy for remote state estimation in the presence of an eavesdropper

We consider a remote state estimation problem in the presence of an eavesdropper over packet dropping links. A smart sensor transmits its local estimates to a legitimate remote estimator, in the course of which an eavesdropper can randomly overhear the transmission. This problem has been well studied for unstable dynamical systems, but seldom for stable systems. In this paper, we target at stable and marginally stable systems and aim to design an event-triggered scheduling strategy by minimizing the expected error covariance at the remote estimator and keeping that at the eavesdropper above a user-specified lower bound. To this end, we model the evolution of the error covariance as an infinite recurrent Markov chain and develop a recurrence relation to describe the stationary distribution of the state at the eavesdropper. Monotonicity and convergence properties of the expected error covariance are further investigated and employed to solve the optimization problem. Numerical examples are provided to validate the theoretical results.

Deep reinforcement learning for scheduling in large-scale networked control systems

This work considers the problem of control and resource scheduling in networked systems. We present DIRA, a Deep reinforcement learning based Iterative Resource Allocation algorithm, which is scalable and control-aware. Our algorithm is tailored towards large-scale problems where control and scheduling need to act jointly to optimize performance. DIRA can be used to schedule general time-domain optimization based controllers. In the present work, we focus on control designs based on suitably adapted linear quadratic regulators. We apply our algorithm to networked systems with correlated fading communication channels. Our simulations show that DIRA scales well to large scheduling problems.

Finite Time Encryption Schedule in the Presence of an Eavesdropper with Operation Cost

In this paper, we consider a remote state estimation problem in the presence of an eavesdropper. A smart sensor takes measurement of a discrete linear time-invariant (LTI) process and sends its local state estimate through a wireless network to a remote estimator. An eavesdropper can overhear the sensor transmissions with a certain probability. To enhance the system privacy level, we propose a novel encryption strategy to minimize a linear combination of the expected error covariance at the remote estimator and the negative of the expected error covariance at the eavesdropper, taking into account the cost of the encryption process. We prove the existence of an optimal deterministic and Markovian policy for such an encryption strategy over a finite time horizon. Two situations, namely, with or without knowledge of the eavesdropper estimation error covariance are studied and the optimal schedule is shown to satisfy the threshold-like structure in both cases. Numerical examples are given to illustrate the results.

Deep Reinforcement Learning for Wireless Sensor Scheduling in Cyber-Physical Systems

In many Cyber-Physical Systems, we encounter the problem of remote state estimation of geographically distributed and remote physical processes. This paper studies the scheduling of sensor transmissions to estimate the states of multiple remote, dynamic processes. Information from the different sensors have to be transmitted to a central gateway over a wireless network for monitoring purposes, where typically fewer wireless channels are available than there are processes to be monitored. For effective estimation at the gateway, the sensors need to be scheduled appropriately, i.e., at each time instant one needs to decide which sensors have network access and which ones do not. To address this scheduling problem, we formulate an associated Markov decision process (MDP). This MDP is then solved using a Deep Q-Network, a recent deep reinforcement learning algorithm that is at once scalable and model-free. We compare our scheduling algorithm to popular scheduling algorithms such as round-robin and reduced-waiting-time, among others. Our algorithm is shown to significantly outperform these algorithms for many example scenarios.

DeepCAS: A Deep Reinforcement Learning Algorithm for Control-Aware Scheduling

We consider networked control systems consisting of multiple independent controlled subsystems, operating over a shared communication network. Such systems are ubiquitous in cyber-physical systems, Internet of Things, and large-scale industrial systems. In many large-scale settings, the size of the communication network is smaller than the size of the system. In consequence, scheduling issues arise. The main contribution of this paper is to develop a deep reinforcement learning-based emph{control-aware} scheduling (textsc{DeepCAS}) algorithm to tackle these issues. We use the following (optimal) design strategy: First, we synthesize an optimal controller for each subsystem; next, we design a learning algorithm that adapts to the chosen subsystems (plants) and controllers. As a consequence of this adaptation, our algorithm finds a schedule that minimizes the emph{control loss}. We present empirical results to show that textsc{DeepCAS} finds schedules with better performance than periodic ones.

Asynchronous stochastic approximations with asymptotically biased errors and deep multi-agent learning

Asynchronous stochastic approximations (SAs) are an important class of model-free algorithms, tools and techniques that are popular in multi-agent and distributed control scenarios. To counter Bellmans curse of dimensionality, such algorithms are coupled with function approximations. Although the learning/ control problem becomes more tractable, function approximations affect stability and convergence. In this paper, we present verifiable sufficient conditions for stability and convergence of asynchronous SAs with biased approximation errors. The theory developed herein is used to analyze Policy Gradient methods and noisy Value Iteration schemes. Specifically, we analyze the asynchronous approximate counterparts of the policy gradient (A2PG) and value iteration (A2VI) schemes. It is shown that the stability of these algorithms is unaffected by biased approximation errors, provided they are asymptotically bounded. With respect to convergence (of A2VI and A2PG), a relationship between the limiting set and the approximation errors is established. Finally, experimental results are presented that support the theory.