TmVO$_4$ exhibits ferroquadrupolar order below 2.15 K with a well-isolated non-Kramers ground state doublet, and is a model system to understand Ising nematic order. We present $^{51}$V nuclear magnetic resonance data as a function of field orientation in a single crystal. Although the spectra are well understood in terms of direct dipolar hyperfine couplings, the spin lattice relaxation rate exhibits strong anisotropy that cannot be understood in terms of magnetic fluctuations. We find that the spin lattice relaxation rate scales with the shear elastic constant associated with the ferroquadrupole phase transition, suggesting that quadrupole (nematic) fluctuations dominate the spin lattice relaxation for in-plane fields.