Nematic states are characterized by rotational symmetry breaking without translational ordering. Recently, nematic superconductivity, in which the superconducting gap spontaneously lifts the rotational symmetry of the lattice, has been discovered. However the pairing mechanism and the mechanism determining the nematic orientation remain unresolved. A first step is to demonstrate control of the nematicity, through application of an external symmetry-breaking field, to determine the sign and strength of coupling to the lattice. Here, we report for the first time control of the nematic orientation of the superconductivity of Sr$_x$Bi$_2$Se$_3$, through externally-applied uniaxial stress. The suppression of subdomains indicates that it is the $Delta_{4y}$ state that is most favoured under compression along the basal Bi-Bi bonds. These results provide an inevitable step towards understanding the microscopic origin of the unique topological nematic superconductivity.