The dispersion relation of surface plasmon polaritons in graphene that includes optical losses is often obtained for complex wave vectors while the frequencies are assumed to be real. This approach, however, is not suitable for describing the temporal dynamics of optical excitations and the spectral properties of graphene. Here, we propose an alternative approach that calculates the dispersion relation in the complex frequency and real wave vector space. This approach provides a clearer insight into the optical properties of a graphene layer and allows us to find the surface plasmon modes of a graphene sheet in the full frequency range, thus removing the earlier reported limitation (1.667 < $hbaromega/mu$ < 2) for the transverse-electric mode. We further develop a simple analytic approximation which accurately describes the dispersion of the surface plasmon polariton modes in graphene. Using this approximation, we show that transverse-electric surface plasmon polaritons propagate along the graphene sheet without losses even at finite temperature.