We investigate the rich quantum phase diagram of Wegners theory of discrete Ising gauge fields interacting with $U(1)$ symmetric single-component fermion matter hopping on a two-dimensional square lattice. In particular limits the model reduces to (i) pure $mathbb{Z}_2$ even and odd gauge theories, (ii) free fermions in a static background of deconfined $mathbb{Z}_2$ gauge fields, (iii) the kinetic Rokhsar-Kivelson quantum dimer model at a generic dimer filling. We develop a local transformation that maps the lattice gauge theory onto a model of $mathbb{Z}_2$ gauge-invariant spin $1/2$ degrees of freedom. Using the mapping, we perform numerical density matrix renormalization group calculations that corroborate our understanding of the limits identified above. Moreover, in the absence of the magnetic plaquette term, we reveal signatures of topologically ordered Dirac semimetal and staggered Mott insulator phases at half-filling. At strong coupling, the lattice gauge theory displays fracton phenomenology with isolated fermions being completely frozen and dimers exhibiting restricted mobility. In that limit, we predict that in the ground state dimers form compact clusters, whose hopping is suppressed exponentially in their size. We determine the band structure of the smallest clusters numerically using exact diagonalization.