If fermionic dark matter (DM) is stabilized by dark $U(1)$ gauge symmetry that is spontaneously broken into its subgroup $Z_2$, the particle contents of the model becomes very rich: DM and excited DM, both of them are Majorana fermions, as well as two dark force mediators, dark photon and dark Higgs boson are naturally present due to the underlying dark gauge symmetry. In this paper, we study the DM bound state formation processes within this scenario, assuming both dark photon and dark Higgs are light mediators and including the effects of excited DM. The Goldstone boson contributions to the potential matrix in the Schr{o}dinger equations are found to be important. The emissions of a longitudinal vector boson (or somehow equivalently a Goldstone boson) during the DM bound state formations are crucial to induce a significant reannihilation process, reducing the dark matter relic abundance. Most of the stringent constraints for this kind of dark matter considered in the literature are simply evaded.