NiGa$_2$S$_4$ is a triangular lattice S=1 system with strong two-dimensionality of the lattice, actively discussed as a candidate to host spin-nematic order brought about by strong quadrupole coupling. Using Raman scattering spectroscopy we identify a phonon of E$_g$ symmetry which can modulate magnetic exchange $J_1$ and produce quadrupole coupling. Additionally, our Raman scattering results demonstrate a loss of local inversion symmetry on cooling, which we associate with sulfur vacancies. This will lead to disordered Dzyaloshinskii-Moriya interactions, which can prevent long range magnetic order. Using magnetic Raman scattering response we identify 160~K as a temperature of an upturn of magnetic correlations. The temperature below 160~K, but above 50~K where antiferromagnetic magnetic start to increase, is a candidate for spin-nematic regime.