We present measurements of the magnetic susceptibility and of the thermal expansion of a LaCoO$_3$ single crystal. Both quantities show a strongly anomalous temperature dependence. Our data are consistently described in terms of a spin-state transition of the Co$^{3+}$ ions with increasing temperature from a low-spin ground state to an intermediate-spin state without (100K - 500K) and with (>500K) orbital degeneracy. We attribute the lack of orbital degeneracy up to 500K to (probably local) Jahn-Teller distortions of the CoO$_6$ octahedra. A strong reduction or disappearance of the Jahn-Teller distortions seems to arise from the insulator-to-metal transition around 500 K.