We study cascaded quadratic soliton compressors and address the physical mechanisms that limit the compression. A nonlocal model is derived, and the nonlocal response is shown to have an additional oscillatory component in the nonstationary regime when the group-velocity mismatch (GVM) is strong. This inhibits efficient compression. Raman-like perturbations from the cascaded nonlinearity, competing cubic nonlinearities, higher-order dispersion, and soliton energy may also limit compression, and through realistic numerical simulations we point out when each factor becomes important. We find that it is theoretically possible to reach the single-cycle regime by compressing high-energy fs pulses for wavelengths $lambda=1.0-1.3 mu{rm m}$ in a $beta$-barium-borate crystal, and it requires that the system is in the stationary regime, where the phase mismatch is large enough to overcome the detrimental GVM effects. However, the simulations show that reaching single-cycle duration is ultimately inhibited by competing cubic nonlinearities as well as dispersive waves, that only show up when taking higher-order dispersion into account.