We report experimental results on the heat conductivity kappa of the S=1/2 spin chain compounds TiOBr and TiOCl for temperatures 5K<T<300K and magnetic fields up to 14. Surprisingly, we find no evidence of a significant magnetic contribution to kappa, which is in stark contrast to recent results on S=1/2 spin chain cuprates. Despite this unexpected result, the thus predominantly phononic heat conductivity of these spin-Peierls compounds exhibits a very unusual behavior. In particular, we observe strong anomalies at the phase transitions Tc1 and Tc2. Moreover, we find an overall but anisotropic suppression of kappa in the intermediate phase which extends even to temperatures higher than Tc2. An external magnetic field causes a slight downshift of the transition at Tc1 and enhances the suppression of kappa up to Tc2. We interprete our findings in terms of strong spin-phonon coupling and phonon scattering arising from spin-driven lattice distortions.