It has been known for decades that thermal conductivity of insulating crystals becomes proportional to the inverse of temperature when the latter is comparable to or higher than the Debye temperature. This behavior has been understood as resulting from Umklapp scattering among phonons. We put under scrutiny the magnitude of the thermal diffusion constant in this regime and find that it does not fall below a threshold set by the square of sound velocity times the Planckian time ($tau_p=hbar/k_BT$). The conclusion, based on scrutinizing the ratio in cubic crystals with high thermal resistivity, appears to hold even in glasses where Umklapp events are not conceivable. Explaining this boundary, reminiscent of a recently-noticed limit for charge transport in metals, is a challenge to theory.