Temporal cavity solitons (CS) are optical pulses that can persist in passive resonators, and they play a key role in the generation of coherent microresonator frequency combs. In resonators made of amorphous materials, such as fused silica, they can exhibit a spectral red-shift due to stimulated Raman scattering. Here we show that this Raman-induced self-frequency-shift imposes a fundamental limit on the duration and bandwidth of temporal CSs. Specifically, we theoretically predict that stimulated Raman scattering introduces a previously unidentified Hopf bifurcation that leads to destabilization of CSs at large pump-cavity detunings, limiting the range of detunings over which they can exist. We have confirmed our theoretical predictions by performing extensive experiments in several different synchronously-driven fiber ring resonators, obtaining results in excellent agreement with numerical simulations. Our results could have significant implications for the future design of Kerr frequency comb systems based on amorphous microresonators.