An archetypical layered topological insulator Bi$_2$Se$_3$ becomes superconductive upon doping with Sr, Nb or Cu. Superconducting properties of these materials in the presence of in-plane magnetic field demonstrate spontaneous symmetry breaking: 180$^circ$-rotation symmetry of superconductivity versus 120$^circ$-rotation symmetry of the crystal. Such behavior brilliantly confirms nematic topological superconductivity. To what extent this nematicity is due to superconducting pairing in these materials, rather than due to crystal structure distortions? This question remained unanswered, because so far no visible deviations from the 3-fold crystal symmetry were resolved in these materials. To address this question we grow high quality single crystals of Sr$_x$Bi$_2$Se$_3$, perform detailed X-ray diffraction and magnetotransport studies and reveal that the observed superconducting nematicity direction correlates with the direction of small structural distortions in these samples( $sim 0.02$% elongation in one crystallographic direction). Additional anisotropy comes from orientation of the crystallite axes. 2-fold symmetry of magnetoresistance observed in the most uniform crystals well above critical temperature demonstrates that these structural distortions are nevertheless strong enough. Our data in combination with strong sample-to-sample variation of the superconductive anisotropy parameter are indicative for significance of the structural factor in the apparent nematic superconductivity in Sr$_x$Bi$_2$Se$_3$.