The ferrimagnetic spinel $mathrm{CoV_2O_4}$ has been a topic of intense recent interest, both as a frustrated insulator with unquenched orbital degeneracy and as a near-itinerant magnet which can be driven metallic with moderate applied pressure. Here, we report on our recent neutron diffraction and inelastic scattering measurements on powders with minimal cation site disorder. Our main new result is the identification of a weak ($frac{Delta a}{a} sim 10^{-4}$), first order structural phase transition at $T^*$ = 90 K, the same temperature where spin canting was seen in recent single crystal measurements. This transition is characterized by a short-range distortion of oxygen octahedral positions, and inelastic data further establish a weak $Deltasim 1.25 meV$ spin gap at low temperature. Together, these findings provide strong support for the local orbital picture and the existence of an orbital glass state at temperatures below $T^*$.