The Virasoro constraints play the important role in the study of matrix models and in understanding of the relation between matrix models and CFTs. Recently the localization calculations in supersymmetric gauge theories produced new families of matrix models and we have very limited knowledge about these matrix models. We concentrate on elliptic generalization of hermitian matrix model which corresponds to calculation of partition function on $S^3 times S^1$ for vector multiplet. We derive the $q$-Virasoro constraints for this matrix model. We also observe some interesting algebraic properties of the $q$-Virasoro algebra.