Band insulators appear in a crystalline system only when the filling -- the number of electrons per unit cell and spin projection -- is an integer. At fractional filling, an insulating phase that preserves all symmetries is a Mott insulator, i.e. it is either gapless or, if gapped, displays fractionalized excitations and topological order. We raise the inverse question -- at an integer filling is a band insulator always possible? Here we show that lattice symmetries may forbid a band insulator even at certain integer fillings, if the crystal is non-symmorphic -- a property shared by a majority of three-dimensional crystal structures. In these cases, one may infer the existence of topological order if the ground state is gapped and fully symmetric. This is demonstrated using a non-perturbative flux threading argument, which has immediate applications to quantum spin systems and bosonic insulators in addition to electronic band structures in the absence of spin-orbit interactions.