Optical measurements in doped Mott insulators have discovered the emergence of spectral weights at mid-infrared (MIR) upon chemical doping and photodoping. MIR weights may have a relation to string-type excitation of spins, which is induced by a doped hole generating misarranged spins with respect to their sublattice. There are two types of string effects: one is an $S^z$ string that is repairable by quantum spin flips and the other is a phase string irreparable by the spin flips. We investigate the effect of $S^{z}$ and phase strings on MIR weights. Calculating the optical conductivity of the single-hole Hubbard model in the strong-coupling regime and the $t$-$J$ model on two-leg ladders by using time-dependent Lanczos and density-matrix renormalization group, we find that phase strings make a crucial effect on the emergence of MIR weights as compared with $S^{z}$ strings. Our findings indicate that a mutual Chern-Simons gauge field acting between spin and charge degrees of freedom, which is the origin of phase strings, is significant for obtaining MIR weights. Conversely, if we remove this gauge field, no phase is picked up by a doped hole. As a result, a spin-polaron accompanied by a local spin distortion emerges and a quasiparticle with a cosine-like energy dispersion is formed in single-particle spectral function. Furthermore, we suggest a Floquet engineering to examine the phase-string effect in cold atoms.