Irradiation-induced lattice defects in silicon carbide (SiC) have already exceeded their previous reputation as purely performance-inhibiting. With their remarkable quantum properties, such as long room-temperature spin coherence and the possibility of downscaling to single-photon source level, they have proven to be promising candidates for a multitude of quantum information applications. One of the most crucial parameters of any quantum system is how long its quantum coherence can be preserved. By using the pulsed optically detected magnetic resonance (ODMR) technique, we investigate the spin-lattice relaxation time ($T_1$) and spin coherence time ($T_2$) of silicon vacancies in 4H-SiC created by neutron, electron and proton irradiation in a broad range of fluences. We also examine the effect of irradiation energy and sample annealing. We establish a robustness of the $T_1$ time against all types of irradiation and reveal a universal scaling of the $T_2$ time with the emitter density. Our results can be used to optimize the coherence properties of silicon vacancy qubits in SiC for specific tasks.