Soft solids like colloidal glasses exhibit a yield stress, above which the system starts to flow. The microscopic analogon in microrheology is the delocalization of a tracer particle subject to an external force exceeding a threshold value, in a glassy host. We characterize this delocalization transition based on a bifurcation analysis of the corresponding mode-coupling theory equations. A schematic model is presented first, that allows analytical progress, and the full physical model is studied numerically next. This analysis yields a continuous type A transition with a critical power law decay of the probe correlation functions with exponent $-1/2$. In order to compare with simulations with a limited duration, a finite time analysis is performed, which yields reasonable results for not-too-small wave vectors. The theoretically predicted findings are verified by Langevin dynamics simulations. For small wave vectors we find anomalous behavior for the probe position correlation function, which can be traced back to a wave vector divergence of the critical amplitude. In addition we propose and test three methods to extract the critical force from experimental data, which provide the same value of the critical force when applied to the finite-time theory or simulations.