Coherent quantum systems are a key resource for emerging quantum technology. Solid-state spin systems are of particular importance for compact and scalable devices. However, interaction with the solid-state host degrades the coherence properties. The negatively-charged silicon vacancy center in diamond is such an example. While spectral properties are outstanding, with optical coherence protected by the defects symmetry, the spin coherence is susceptible to rapid orbital relaxation limiting the spin dephasing time. A prolongation of the orbital relaxation time is therefore of utmost urgency and has been tackled by operating at very low temperatures or by introducing large strain. However, both methods have significant drawbacks, the former requires use of dilution refrigerators and the latter affects intrinsic symmetries. Here, a novel method is presented to prolong the orbital relaxation with a locally modified phonon density of states in the relevant frequency range, by restricting the diamond host to below 100 nm. The method works at liquid Helium temperatures of few Kelvin and in the low-strain regime.