We perform hole burning with a low drift stabilized laser within the zero phonon line of the 4f-5d transition in Ce$^{3+}$:Y$_2$SiO$_5$ at 2K. The narrowest spectral holes appear for small applied magnetic fields and are $6pm4$ MHz wide (FWHM). This puts an upper bound on the homogeneous linewidth of the transition to $3pm2$ MHz, which is close to lifetime limited. The spin level relaxation time is measured to $72pm21$ ms with a magnetic field of 10 mT. A slow permanent hole burning mechanism is observed. If the excitation frequency is not changed the fluorescence intensity is reduced by more than 50$%$ after a couple of minutes of continuous excitation. The spectral hole created by the permanent hole burning has a width in the tens of MHz range, which indicates that a trapping mechanism occurs via the 5d-state.