A bound state between a quantum emitter (QE) and surface plasmon polaritons (SPPs) can be formed, where the QE is partially stabilized in its excited state. We put forward a general approach for calculating the energy level shift at a negative frequency $omega$, which is just the negative of the nonresonant part for the energy level shift at positive frequency $-omega$. We also propose an efficient formalism for obtaining the long-time value of the excited-state population without calculating the eigenfrequency of the bound state or performing a time evolution of the system, in which the probability amplitude for the excited state in the steady limit is equal to one minus the integral of the evolution spectrum over the positive frequency range. With the above two quantities obtained, we show that the non-Markovian decay dynamics in the presence of a bound state can be obtained by the method based on the Greens function expression for the evolution operator. A general criterion for identifying the existence of a bound state is presented. These are numerically demonstrated for a QE located around a nanosphere and in a gap plasmonic nanocavity. These findings are instructive in the fields of coherent light-matter interactions.