We perform ultrasound velocity measurements on a single crystal of nearly-metallic spinel Co$_{1.21}$V$_{1.79}$O$_4$ which exhibits a ferrimagnetic phase transition at $T_C sim$ 165 K. The experiments reveal a variety of elastic anomalies in not only the paramagnetic phase above $T_C$ but also the ferrimagnetic phase below $T_C$, which should be driven by the nearly-itinerant character of the orbitally-degenerate V 3$d$ electrons. In the paramagnetic phase above $T_C$, the elastic moduli exhibit elastic-mode-dependent unusual temperature variations, suggesting the existence of a dynamic spin-cluster state. Furthermore, above $T_C$, the sensitive magnetic-field response of the elastic moduli suggests that, with the negative magnetoresistance, the magnetic-field-enhanced nearly-itinerant character of the V 3$d$ electrons emerges from the spin-cluster state. This should be triggered by the inter-V-site interactions acting on the orbitally-degenerate 3$d$ electrons. In the ferrimagnetic phase below $T_C$, the elastic moduli exhibit distinct anomalies at $T_1sim$ 95 K and $T_2sim$ 50 K, with a sign change of the magnetoresistance at $T_1$ (positive below $T_1$) and an enhancement of the positive magnetoresistance below $T_2$, respectively. These observations below $T_C$ suggest the successive occurrence of an orbital glassy order at $T_1$ and a structural phase transition at $T_2$, where the rather localized character of the V 3$d$ electrons evolves below $T_1$ and is further enhanced below $T_2$.