Lasers based on Cr$^{2+}$-doped II-VI material, often known as the Ti:Sapphire of the mid-infrared, can directly provide few-cycle pulses with super-octave-spanning spectra, and serve as efficient drivers for generating broadband mid-infrared radiation. It is expected that the wider adoption of this technology benefits from more compact and cost-effective embodiments. Here, we report the first directly diode-pumped, Kerr-lens mode-locked Cr$^{2+}$-doped II-VI oscillator pumped by a single InP diode, providing average powers of over 500 mW and pulse durations of 45 fs - shorter than six optical cycles at 2.4 $mu$m. These correspond to a sixty-fold increase in peak power compared to the previous diode-pumped record, and are at similar levels with respect to more mature fiber-pumped oscillators. The diode-pumped femtosecond oscillator presented here constitutes a key step towards a more accessible alternative to synchrotron-like infrared radiation, and is expected to accelerate research in laser spectroscopy and ultrafast infrared optics.