The combination of electronic correlation and spin-orbit coupling is thought to precipitate a variety of highly unusual electronic phases in solids, including topological and quantum spin liquid states. We report a Raman scattering study that provides evidence for unconventional excitations in $alpha$-RuCl$_3$, a spin-orbit coupled Mott insulator on the honeycomb lattice. In particular, our measurements reveal unusual magnetic scattering, typified by a broad continuum. The temperature dependence of this continuum is evident over a large scale compared to the magnetic ordering temperature, suggestive of frustrated magnetic interactions. This is confirmed through an analysis of the phonon linewidths, which show a related anomaly due to spin-phonon coupling. These observations are in line with theoretical expectations for the Heisenberg-Kitaev model and suggest that $alpha$-RuCl$_3$ may be close to a quantum spin liquid ground state.