A toggle switch consists of two genes that mutually repress each other. This regulatory motif is active during cell differentiation and is thought to act as a memory device, being able to choose and maintain cell fate decisions. In this contribution, we study the stability and dynamics of a two-stage gene expression switch within a probabilistic framework inspired by the properties of the Pu/Gata toggle switch in myeloid progenitor cells. We focus on low mRNA numbers, high protein abundance and monomeric transcription factor binding. Contrary to the expectation from a deterministic description, this switch shows complex multi-attractor dy- namics without autoactivation and cooperativity. Most importantly, the four attractors of the system, which only emerge in a probabilistic two-stage description, can be identified with committed and primed states in cell differentiation. We first study the dynamics of the system and infer the mechanisms that move the system between attractors using both the quasi-potential and the probability flux of the system. Second, we show that the residence times of the system in one of the committed attractors are geometrically distributed and provide an analytical expression of the distribution. Most importantly we find that the mean residence time increases linearly with the mean protein level. Finally, we study the implications of this distribution for the stability of a switch and discuss the influence of the stability on a specific cell differentiation mechanism. Our model explains lineage priming and proposes the need of either high protein numbers or long term modifications such as chromatin remodeling to achieve stable cell fate decisions. Notably we present a system with high protein abundance that nevertheless requires a probabilistic description to exhibit multistability, complex switching dynamics and lineage priming.