Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userTowards quantifying information flows: relative entropy in deep neural networks and the renormalization group
146
0
0.0
(
0
)
Ask ChatGPT about the research
No Arabic abstract
We investigate the analogy between the renormalization group (RG) and deep neural networks, wherein subsequent layers of neurons are analogous to successive steps along the RG. In particular, we quantify the flow of information by explicitly computing the relative entropy or Kullback-Leibler divergence in both the one- and two-dimensional Ising models under decimation RG, as well as in a feedforward neural network as a function of depth. We observe qualitatively identical behavior characterized by the monotonic increase to a parameter-dependent asymptotic value. On the quantum field theory side, the monotonic increase confirms the connection between the relative entropy and the c-theorem. For the neural networks, the asymptotic behavior may have implications for various information maximization methods in machine learning, as well as for disentangling compactness and generalizability. Furthermore, while both the two-dimensional Ising model and the random neural networks we consider exhibit non-trivial critical points, the relative entropy appears insensitive to the phase structure of either system. In this sense, more refined probes are required in order to fully elucidate the flow of information in these models.
rate research
Read More
Quantum Renyi relative entropies provide a one-parameter family of distances between density matrices, which generalizes the relative entropy and the fidelity. We study these measures for renormalization group flows in quantum field theory. We derive explicit expressions in free field theory based on the real time approach. Using monotonicity properties, we obtain new inequalities that need to be satisfied by consistent renormalization group trajectories in field theory. These inequalities play the role of a second law of thermodynamics, in the context of renormalization group flows. Finally, we apply these results to a tractable Kondo model, where we evaluate the Renyi relative entropies explicitly. An outcome of this is that Andersons orthogonality catastrophe can be avoided by working on a Cauchy surface that approaches the light-cone.
The necessary information to distinguish a local inhomogeneous mass density field from its spatial average on a compact domain of the universe can be measured by relative information entropy. The Kullback-Leibler (KL) formula arises very naturally in this context, however, it provides a very complicated way to compute the mutual information between spatially separated but causally connected regions of the universe in a realistic, inhomogeneous model. To circumvent this issue, by considering a parametric extension of the KL measure, we develop a simple model to describe the mutual information which is entangled via the gravitational field equations. We show that the Tsallis relative entropy can be a good approximation in the case of small inhomogeneities, and for measuring the independent relative information inside the domain, we propose the Renyi relative entropy formula.
We consider line defects in d-dimensional Conformal Field Theories (CFTs). The ambient CFT places nontrivial constraints on Renormalization Group (RG) flows on such line defects. We show that the flow on line defects is consequently irreversible and furthermore a canonical decreasing entropy function exists. This construction generalizes the g theorem to line defects in arbitrary dimensions. We demonstrate our results in a flow between Wilson loops in 4 dimensions.
We apply the method of graphical functions that was recently extended to six dimensions for scalar theories, to $phi^3$ theory and compute the $beta$ function, the wave function anomalous dimension as well as the mass anomalous dimension in the $overline{mbox{MS}}$ scheme to five loops. From the results we derive the corresponding renormalization group functions for the Lee-Yang edge singularity problem and percolation theory. After determining the $varepsilon$ expansions of the respective critical exponents to $mathcal{O}(varepsilon^5)$ we apply recent resummation technology to obtain improved exponent estimates in 3, 4 and 5 dimensions. These compare favourably with estimates from fixed dimension numerical techniques and refine the four loop results. To assist with this comparison we collated a substantial amount of data from numerical techniques which are included in tables for each exponent.
We introduce a general method for optimizing real-space renormalization-group transformations to study the critical properties of a classical system. The scheme is based on minimizing the Kullback-Leibler divergence between the distribution of the system and the normalized normalizing factor of the transformation parametrized by a restricted Boltzmann machine. We compute the thermal critical exponent of the two-dimensional Ising model using the trained optimal projector and obtain a very accurate thermal critical exponent $y_t=1.0001(11)$ after the first step of the transformation.
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
