Sr$_x$Bi$_2$Se$_3$ and the related compounds Cu$_x$Bi$_2$Se$_3$ and Nb$_x$Bi$_2$Se$_3$ have attracted considerable interest, as these materials may be realizations of unconventional topological superconductors. Superconductivity with T$_c$ ~ 3 K in Sr$_x$Bi$_2$Se$_3$ arises upon intercalation of Sr into the layered topological insulator Bi$_2$Se$_3$. Here we elucidate the anisotropy of the normal and superconducting state of Sr$_{0.1}$Bi$_2$Se$_3$ with angular dependent magnetotransport and thermodynamic measurements. High resolution x-ray diffraction studies underline the high crystalline quality of the samples. We demonstrate that the normal state electronic and magnetic properties of Sr$_{0.1}$Bi$_2$Se$_3$ are isotropic in the basal plane while we observe a large two-fold in-plane anisotropy of the upper critical field in the superconducting state. Our results support the recently proposed odd-parity nematic state characterized by a nodal gap of $E_u$ symmetry in Sr$_x$Bi$_2$Se$_3$.