A new definition of continuous-time equilibrium controls is introduced. As opposed to the standard definition, which involves a derivative-type operation, the new definition parallels how a discrete-time equilibrium is defined, and allows for unambiguous economic interpretation. The terms strong equilibria and weak equilibria are coined for controls under the new and the standard definitions, respectively. When the state process is a time-homogeneous continuous-time Markov chain, a careful asymptotic analysis gives complete characterizations of weak and strong equilibria. Thanks to Kakutani-Fans fixed-point theorem, general existence of weak and strong equilibria is also established, under additional compactness assumption. Our theoretic results are applied to a two-state model under non-exponential discounting. In particular, we demonstrate explicitly that there can be incentive to deviate from a weak equilibrium, which justifies the need for strong equilibria. Our analysis also provides new results for the existence and characterization of discrete-time equilibria under infinite horizon.