We provide a general framework for studying recurrent neural networks (RNNs) trained by injecting noise into hidden states. Specifically, we consider RNNs that can be viewed as discretizations of stochastic differential equations driven by input data
. This framework allows us to study the implicit regularization effect of general noise injection schemes by deriving an approximate explicit regularizer in the small noise regime. We find that, under reasonable assumptions, this implicit regularization promotes flatter minima; it biases towards models with more stable dynamics; and, in classification tasks, it favors models with larger classification margin. Sufficient conditions for global stability are obtained, highlighting the phenomenon of stochastic stabilization, where noise injection can improve stability during training. Our theory is supported by empirical results which demonstrate improved robustness with respect to various input perturbations, while maintaining state-of-the-art performance.
Recurrent neural networks (RNNs) are brain-inspired models widely used in machine learning for analyzing sequential data. The present work is a contribution towards a deeper understanding of how RNNs process input signals using the response theory fr
om nonequilibrium statistical mechanics. For a class of continuous-time stochastic RNNs (SRNNs) driven by an input signal, we derive a Volterra type series representation for their output. This representation is interpretable and disentangles the input signal from the SRNN architecture. The kernels of the series are certain recursively defined correlation functions with respect to the unperturbed dynamics that completely determine the output. Exploiting connections of this representation and its implications to rough paths theory, we identify a universal feature -- the response feature, which turns out to be the signature of tensor product of the input signal and a natural support basis. In particular, we show that SRNNs, with only the weights in the readout layer optimized and the weights in the hidden layer kept fixed and not optimized, can be viewed as kernel machines operating on a reproducing kernel Hilbert space associated with the response feature.
We study functionals, such as heat and work, along trajectories of a class of multi-dimensional generalized Langevin systems in various limiting situations that correspond to different level of homogenization. These are the situations where one or mo
re of the inertial time scale(s), the memory time scale(s) and the noise correlation time scale(s) of the systems are taken to zero. We find that, unless one restricts to special situations that do not break symmetry of the Onsager matrix associated with the fast dynamics, it is generally not possible to express the effective evolution of these functionals solely in terms of trajectory of the homogenized process describing the system dynamics via the widely adopted Stratonovich convention. In fact, an anomalous term is often needed for a complete description, implying that convergence of these functionals needs more information than simply the limit of the dynamical process. We trace the origin of such impossibility to area anomaly, thereby linking the symmetry breaking and area anomaly. This hold important consequences for many nonequilibrium systems that can be modeled by generalized Langevin equations. Our convergence results hold in a strong pathwise sense.
We study the problem of predicting rare critical transition events for a class of slow-fast nonlinear dynamical systems. The state of the system of interest is described by a slow process, whereas a faster process drives its evolution and induces cri
tical transitions. By taking advantage of recent advances in reservoir computing, we present a data-driven method to predict the future evolution of the state. We show that our method is capable of predicting a critical transition event at least several numerical time steps in advance. We demonstrate the success as well as the limitations of our method using numerical experiments on three examples of systems, ranging from low dimensional to high dimensional. We discuss the mathematical and broader implications of our results.
In stochastic thermodynamics standard concepts from macroscopic thermodynamics, such as heat, work, and entropy production, are generalized to small fluctuating systems by defining them on a trajectory-wise level. In Langevin systems with continuous
state-space such definitions involve stochastic integrals along system trajectories, whose specific values depend on the discretization rule used to evaluate them (i.e. the interpretation of the noise terms in the integral). Via a systematic mathematical investigation of this apparent dilemma, we corroborate the widely used standard interpretation of heat- and work-like functionals as Stratonovich integrals. We furthermore recapitulate the anomalies that are known to occur for entropy production in the presence of temperature gradients.
We study homogenization for a class of generalized Langevin equations (GLEs) with state-dependent coefficients and exhibiting multiple time scales. In addition to the small mass limit, we focus on homogenization limits, which involve taking to zero t
he inertial time scale and, possibly, some of the memory time scales and noise correlation time scales. The latter are meaningful limits for a class of GLEs modeling anomalous diffusion. We find that, in general, the limiting stochastic differential equations (SDEs) for the slow degrees of freedom contain non-trivial drift correction terms and are driven by non-Markov noise processes. These results follow from a general homogenization theorem stated and proven here. We illustrate them using stochastic models of particle diffusion.
We study the small mass limit (or: the Smoluchowski-Kramers limit) of a class of quantum Brownian motions with inhomogeneous damping and diffusion. For Ohmic bath spectral density with a Lorentz-Drude cutoff, we derive the Heisenberg-Langevin equatio
ns for the particles observables using a quantum stochastic calculus approach. We set the mass of the particle to equal $m = m_{0} epsilon$, the reduced Planck constant to equal $hbar = epsilon$ and the cutoff frequency to equal $Lambda = E_{Lambda}/epsilon$, where $m_0$ and $E_{Lambda}$ are positive constants, so that the particles de Broglie wavelength and the largest energy scale of the bath are fixed as $epsilon to 0$. We study the limit as $epsilon to 0$ of the rescaled model and derive a limiting equation for the (slow) particles position variable. We find that the limiting equation contains several drift correction terms, the quantum noise-induced drifts, including terms of purely quantum nature, with no classical counterparts.
We study the dynamics of a quantum impurity immersed in a Bose-Einstein condensate as an open quantum system in the framework of the quantum Brownian motion model. We derive a generalized Langevin equation for the position of the impurity. The Langev
in equation is an integrodifferential equation that contains a memory kernel and is driven by a colored noise. These result from considering the environment as given by the degrees of freedom of the quantum gas, and thus depend on its parameters, e.g. interaction strength between the bosons, temperature, etc. We study the role of the memory on the dynamics of the impurity. When the impurity is untrapped, we find that it exhibits a super-diffusive behavior at long times. We find that back-flow in energy between the environment and the impurity occurs during evolution. When the particle is trapped, we calculate the variance of the position and momentum to determine how they compare with the Heisenberg limit. One important result of this paper is that we find position squeezing for the trapped impurity at long times. We determine the regime of validity of our model and the parameters in which these effects can be observed in realistic experiments.
We study a class of systems whose dynamics are described by generalized Langevin equations with state-dependent coefficients. We find that in the limit, in which all the characteristic time scales vanish at the same rate, the position variable of the
system converges to a homogenized process, described by an equation containing additional drift terms induced by the noise. The convergence results are obtained using the main result in cite{hottovy2015smoluchowski}, whose version is proven here under a weaker spectral assumption on the damping matrix. We apply our results to study thermophoresis of a Brownian particle in a non-equilibrium heat bath.
The theory of quantum Brownian motion describes the properties of a large class of open quantum systems. Nonetheless, its description in terms of a Born-Markov master equation, widely used in the literature, is known to violate the positivity of the
density operator at very low temperatures. We study an extension of existing models, leading to an equation in the Lindblad form, which is free of this problem. We study the dynamics of the model, including the detailed properties of its stationary solution, for both constant and position-dependent coupling of the Brownian particle to the bath, focusing in particular on the correlations and the squeezing of the probability distribution induced by the environment
