Transition metal dichalcogenide MoTe$_2$ is an important candidate for realizing the newly predicted type-IIWeyl fermions, for which the breaking of the inversion symmetry is a prerequisite. Here we present direct spectroscopic evidence for the inversion symmetry breaking in the low temperature phase of MoTe$_2$ by systematic Raman experiments and first principles calculations. We identify five lattice vibrational modes which are Raman active only in noncentrosymmetric structure at low temperature. A hysteresis is also observed in the peak intensity of inversion symmetry activated Raman modes, confirming a temperature induced structural phase transition with a concomitant change in the inversion symmetry. Our results provide definitive evidence for the low temperature noncentrosymmetric T$_d$ phase from vibrational spectroscopy, and suggest MoTe$_2$ as an ideal candidate for investigating the temperature induced topological phase transition.