Active plasma lensing is a promising technology for compact focusing of particle beams that has seen a recent surge of interest. While these lenses can provide strong focusing gradients of order kT/m and focusing in both transverse planes, there are limitations from nonlinear aberrations, causing emittance growth in the beams being focused. One cause of such aberrations is beam-driven plasma wakefields, present if the beam density is sufficiently high. We develop simple, but powerful analytic formulas for the effective focusing gradient from these wakefields, and use this to set limits on which parts of the beam and plasma parameter space permits distortion-free use of active plasma lenses. It is concluded that the application of active plasma lenses to conventional and plasma-based linear colliders may prove very challenging, except perhaps in the final focus system, unless the typical discharge currents used are dramatically increased, and that in general these lenses are better suited for accelerator applications with lower beam intensities.