We report on an experimental study of the Lipkin-Meshkov-Glick model of quantum spins interacting at infinite range in a transverse magnetic field, which exhibits a ferromagnetic phase transition in the thermodynamic limit. We use Dysprosium atoms of electronic spin $J=8$, subjected to a quadratic Zeeman light shift, to simulate $2J=16$ interacting spins $1/2$. We probe the system microscopically using single magnetic sublevel resolution, giving access to the spin projection parity, which is the collective observable characterizing the underlying $mathbb{Z}_2$ symmetry. We measure the thermodynamic properties and dynamical response of the system, and study the quantum critical behavior around the transition point. In the ferromagnetic phase, we achieve coherent tunneling between symmetry-broken states, and test the link between symmetry breaking and the appearance of a finite order parameter.