In this paper, we have systematically studied the single hole problem in two-leg Hubbard and $t$-$J$ ladders by large-scale density-matrix renormalization group calculations. We found that the doped hole in both models behaves similarly with each other while the three-site correlated hopping term is not important in determining the ground state properties. For more insights, we have also calculated the elementary excitations, i.e., the energy gaps to the excited states of the system. In the strong rung limit, we found that the doped hole behaves as a Bloch quasiparticle in both systems where the spin and charge of the doped hole are tightly bound together. In the isotropic limit, while the hole still behaves like a quasiparticle in the long-wavelength limit, its spin and charge components are only loosely bound together with a nontrivial mutual statistics inside the quasiparticle. Our results show that this mutual statistics can lead to an important residual effect which dramatically changes the local structure of the ground state wavefunction.