The mutual information of a single-layer perceptron with $N$ Gaussian inputs and $P$ deterministic binary outputs is studied by numerical simulations. The relevant parameters of the problem are the ratio between the number of output and input units, $alpha = P/N$, and those describing the two-point correlations between inputs. The main motivation of this work refers to the comparison between the replica computation of the mutual information and an analytical solution valid up to $alpha sim O(1)$. The most relevant results are: (1) the simulation supports the validity of the analytical prediction, and (2) it also verifies a previously proposed conjecture that the replica solution interpolates well between large and small values of $alpha$.
Download