We implement three-dimensional polarization gradient cooling of trapped ions. Counter-propagating laser beams near $393,$nm impinge in lin$,perp,$lin configuration, at a frequency below the S$_{1/2}$ to P$_{3/2}$ resonance in $^{40}$Ca$^+$. We demonstrate mean phonon numbers of $5.4(4)$ at a trap frequency of $2pi times 285,$kHz and $3.3(4)$ at $2pitimes480,$kHz, in the axial and radial directions, respectively. Our measurements demonstrate that cooling with laser beams detuned to lower frequencies from the resonance is robust against an elevated phonon occupation number, and thus works well for an initial ion motion far out of the Lamb-Dicke regime, for up to four ions, and for a micromotion modulation index $betaleq 0.1$. Still, we find that the spectral impurity of the laser field influences both, cooling rates and cooling limits. Thus, a Fabry-P{e}rot cavity filter is employed for efficiently suppressing amplified spontaneous emission of the diode laser.