Recently it was demonstrated that Sr intercalation provides a new route to induce superconductivity in the topological insulator Bi$_2$Se$_3$. Topological superconductors are predicted to be unconventional, with mixed even and odd parity Cooper pairs states. An adequate probe to test for unconventional superconductivity is the upper critical field, $B_{c2}$. For a standard BCS layered superconductor $B_{c2}$ shows an anisotropy when the magnetic field is applied parallel and perpendicular to the layers, but is isotropic when the field is rotated in the plane of the layers. Here we report measurements of the upper critical field of superconducting Sr$_x$Bi$_2$Se$_3$ crystals ($T_c = 3.0$~K). Surprisingly, field-angle dependent magnetotransport measurements reveal a large anisotropy of $B_{c2}$ when the magnet field is rotated in the basal plane. The large two-fold anisotropy, while six-fold is anticipated, cannot be explained with the Ginzburg-Landau anisotropic effective mass model or flux flow induced by the Lorentz force. The rotational symmetry breaking of $B_{c2}$ indicates unconventional superconductivity with an odd-parity polarized triplet Cooper pair state ($Delta_4$-pairing) recently proposed for rhombohedral topological superconductors, or might have a structural nature, such as self-organized stripe ordering of Sr atoms.