Subscribe to the gold package and get unlimited access to Shamra Academy
Register a new userBayesian feature selection with strongly-regularizing priors maps to the Ising Model
120
0
0.0
(
0
)
Ask ChatGPT about the research
No Arabic abstract
Identifying small subsets of features that are relevant for prediction and/or classification tasks is a central problem in machine learning and statistics. The feature selection task is especially important, and computationally difficult, for modern datasets where the number of features can be comparable to, or even exceed, the number of samples. Here, we show that feature selection with Bayesian inference takes a universal form and reduces to calculating the magnetizations of an Ising model, under some mild conditions. Our results exploit the observation that the evidence takes a universal form for strongly-regularizing priors --- priors that have a large effect on the posterior probability even in the infinite data limit. We derive explicit expressions for feature selection for generalized linear models, a large class of statistical techniques that include linear and logistic regression. We illustrate the power of our approach by analyzing feature selection in a logistic regression-based classifier trained to distinguish between the letters B and D in the notMNIST dataset.
rate research
Read More
Feature selection, identifying a subset of variables that are relevant for predicting a response, is an important and challenging component of many methods in statistics and machine learning. Feature selection is especially difficult and computationally intensive when the number of variables approaches or exceeds the number of samples, as is often the case for many genomic datasets. Here, we introduce a new approach -- the Bayesian Ising Approximation (BIA) -- to rapidly calculate posterior probabilities for feature relevance in L2 penalized linear regression. In the regime where the regression problem is strongly regularized by the prior, we show that computing the marginal posterior probabilities for features is equivalent to computing the magnetizations of an Ising model. Using a mean field approximation, we show it is possible to rapidly compute the feature selection path described by the posterior probabilities as a function of the L2 penalty. We present simulations and analytical results illustrating the accuracy of the BIA on some simple regression problems. Finally, we demonstrate the applicability of the BIA to high dimensional regression by analyzing a gene expression dataset with nearly 30,000 features.
I propose a variational approach to maximum pseudolikelihood inference of the Ising model. The variational algorithm is more computationally efficient, and does a better job predicting out-of-sample correlations than $L_2$ regularized maximum pseudolikelihood inference as well as mean field and isolated spin pair approximations with pseudocount regularization. The key to the approach is a variational energy that regularizes the inference problem by shrinking the couplings towards zero, while still allowing some large couplings to explain strong correlations. The utility of the variational pseudolikelihood approach is illustrated by training an Ising model to represent the letters A-J using samples of letters from different computer fonts.
We study in detail the active Ising model, a stochastic lattice gas where collective motion emerges from the spontaneous breaking of a discrete symmetry. On a 2d lattice, active particles undergo a diffusion biased in one of two possible directions (left and right) and align ferromagnetically their direction of motion, hence yielding a minimal flocking model with discrete rotational symmetry. We show that the transition to collective motion amounts in this model to a bona fide liquid-gas phase transition in the canonical ensemble. The phase diagram in the density/velocity parameter plane has a critical point at zero velocity which belongs to the Ising universality class. In the density/temperature canonical ensemble, the usual critical point of the equilibrium liquid-gas transition is sent to infinite density because the different symmetries between liquid and gas phases preclude a supercritical region. We build a continuum theory which reproduces qualitatively the behavior of the microscopic model. In particular we predict analytically the shapes of the phase diagrams in the vicinity of the critical points, the binodal and spinodal densities at coexistence, and the speeds and shapes of the phase-separated profiles.
In this paper, we introduce a new methodology for Bayesian variable selection in linear regression that is independent of the traditional indicator method. A diagonal matrix $mathbf{G}$ is introduced to the prior of the coefficient vector $boldsymbol{beta}$, with each of the $g_j$s, bounded between $0$ and $1$, on the diagonal serves as a stabilizer of the corresponding $beta_j$. Mathematically, a promising variable has a $g_j$ value that is close to $0$, whereas the value of $g_j$ corresponding to an unpromising variable is close to $1$. This property is proven in this paper under orthogonality together with other asymptotic properties. Computationally, the sample path of each $g_j$ is obtained through Metropolis-within-Gibbs sampling method. Also, in this paper we give two simulations to verify the capability of this methodology in variable selection.
We study the spin-$1/2$ Ising chain with multispin interactions $K$ involving the product of $m$ successive spins, for general values of $m$. Using a change of spin variables the zero-field partition function of a finite chain is obtained for free and periodic boundary conditions (BC) and we calculate the two-spin correlation function. When placed in an external field $H$ the system is shown to be self-dual. Using another change of spin variables the one-dimensional (1D) Ising model with multispin interactions in a field is mapped onto a zero-field rectangular Ising model with first-neighbour interactions $K$ and $H$. The 2D system, with size $mtimes N/m$, has the topology of a cylinder with helical BC. In the thermodynamic limit $N/mtoinfty$, $mtoinfty$, a 2D critical singularity develops on the self-duality line, $sinh 2Ksinh 2H=1$.
suggested questions
Log in to be able to interact and post comments
comments
Fetching comments
Sign in to be able to follow your search criteria
