Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userLearning Maximum Entropy Models from finite size datasets: a fast Data-Driven algorithm allows sampling from the posterior distribution
121
0
0.0
(
0
)
Authors
Ulisse Ferrari
Ask ChatGPT about the research
No Arabic abstract
Maximum entropy models provide the least constrained probability distributions that reproduce statistical properties of experimental datasets. In this work we characterize the learning dynamics that maximizes the log-likelihood in the case of large but finite datasets. We first show how the steepest descent dynamics is not optimal as it is slowed down by the inhomogeneous curvature of the model parameters space. We then provide a way for rectifying this space which relies only on dataset properties and does not require large computational efforts. We conclude by solving the long-time limit of the parameters dynamics including the randomness generated by the systematic use of Gibbs sampling. In this stochastic framework, rather than converging to a fixed point, the dynamics reaches a stationary distribution, which for the rectified dynamics reproduces the posterior distribution of the parameters. We sum up all these insights in a rectified Data-Driven algorithm that is fast and by sampling from the parameters posterior avoids both under- and over-fitting along all the directions of the parameters space. Through the learning of pairwise Ising models from the recording of a large population of retina neurons, we show how our algorithm outperforms the steepest descent method.
rate research
Read More
Many real-world mission-critical applications require continual online learning from noisy data and real-time decision making with a defined confidence level. Probabilistic models and stochastic neural networks can explicitly handle uncertainty in data and allow adaptive learning-on-the-fly, but their implementation in a low-power substrate remains a challenge. Here, we introduce a novel hardware fabric that implements a new class of stochastic NN called Neural-Sampling-Machine that exploits stochasticity in synaptic connections for approximate Bayesian inference. Harnessing the inherent non-linearities and stochasticity occurring at the atomic level in emerging materials and devices allows us to capture the synaptic stochasticity occurring at the molecular level in biological synapses. We experimentally demonstrate in-silico hybrid stochastic synapse by pairing a ferroelectric field-effect transistor -based analog weight cell with a two-terminal stochastic selector element. Such a stochastic synapse can be integrated within the well-established crossbar array architecture for compute-in-memory. We experimentally show that the inherent stochastic switching of the selector element between the insulator and metallic state introduces a multiplicative stochastic noise within the synapses of NSM that samples the conductance states of the FeFET, both during learning and inference. We perform network-level simulations to highlight the salient automatic weight normalization feature introduced by the stochastic synapses of the NSM that paves the way for continual online learning without any offline Batch Normalization. We also showcase the Bayesian inferencing capability introduced by the stochastic synapse during inference mode, thus accounting for uncertainty in data. We report 98.25%accuracy on standard image classification task as well as estimation of data uncertainty in rotated samples.
We study the typical (median) value of the minimum gap in the quantum version of the Exact Cover problem using Quantum Monte Carlo simulations, in order to understand the complexity of the quantum adiabatic algorithm (QAA) for much larger sizes than before. For a range of sizes, N <= 128, where the classical Davis-Putnam algorithm shows exponential median complexity, the QAA shows polynomial median complexity. The bottleneck of the algorithm is an isolated avoided crossing point of a Landau-Zener type (collision between the two lowest energy levels only).
Quantum phase transitions are usually observed in ground states of correlated systems. Remarkably, eigenstate phase transitions can also occur at finite energy density in disordered, isolated quantum systems. Such transitions fall outside the framework of statistical mechanics as they involve the breakdown of ergodicity. Here, we consider what general constraints can be imposed on the nature of eigenstate transitions due to the presence of disorder. We derive Harris-type bounds on the finite-size scaling exponents of the mean entanglement entropy and level statistics at the many-body localization phase transition using several different arguments. Our results are at odds with recent small-size numerics, for which we estimate the crossover scales beyond which the Harris bound must hold.
We have investigated the phase transition in the Heisenberg spin glass using massive numerical simulations to study larger sizes, 48x48x48, than have been attempted before at a spin glass phase transition. A finite-size scaling analysis indicates that the data is compatible with the most economical scenario: a common transition temperature for spins and chiralities.
Image denoising is a well-known and well studied problem, commonly targeting a minimization of the mean squared error (MSE) between the outcome and the original image. Unfortunately, especially for severe noise levels, such Minimum MSE (MMSE) solutions may lead to blurry output images. In this work we propose a novel stochastic denoising approach that produces viable and high perceptual quality results, while maintaining a small MSE. Our method employs Langevin dynamics that relies on a repeated application of any given MMSE denoiser, obtaining the reconstructed image by effectively sampling from the posterior distribution. Due to its stochasticity, the proposed algorithm can produce a variety of high-quality outputs for a given noisy input, all shown to be legitimate denoising results. In addition, we present an extension of our algorithm for handling the inpainting problem, recovering missing pixels while removing noise from partially given data.
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
