(abridged) We studied the energy and frequency dependence of the Fourier time lags and intrinsic coherence of the kHz QPOs in the NS LMXBs 4U 1608-52 and 4U 1636-53 using RXTE data. In both sources we confirmed energy-dependent soft lags of 10-100 mu s for the lower kHz QPO. We also found that the time lags of the upper kHz QPO are independent of energy and inconsistent with the soft lags of the lower kHz QPO. The intrinsic coherence of the lower kHz QPO remains constant at 0.6 from 5 to 12 keV, and then drops to zero, while for the upper kHz QPO the intrinsic coherence is consistent with zero across the full energy range. The intrinsic coherence of the upper kHz QPO is consistent with zero over the full frequency range of the QPO, except in 4U 1636-53 at ~780 Hz where it increases to 0.13. In 4U 1636-53, for the lower kHz QPO the 4-12 keV photons lag the 12-20 keV ones by 25 mu s in the QPO frequency range 500-850 Hz, with the lags decreasing to 15 mu s at higher frequencies. In 4U 1608-52 the soft lags of the lower kHz QPO remain constant at 40 mu s. In 4U 1636-53, for the upper kHz QPO the 12-20 keV photons lag the 4-12 keV ones by 11 +/- 3 mu s, independent of QPO frequency; we found consistent results for the time lags of the upper kHz QPO in 4U 1608-52. The intrinsic coherence of the lower kHz QPO increases from ~0-0.4 at 600 Hz to 1 and 0.6 at 800 Hz in 4U 1636-53 and 4U 1608-52, respectively. In 4U 1636-53 it decreases to 0.5 at 920 Hz, while in 4U 1608-52 we do not have data above 800 Hz. We discuss our results in the context of scenarios in which the soft lags are either due to reflection off the accretion disc or up-/down-scattering in a hot medium close to the neutron star. We finally explore the connection between, on one hand the time lags and the intrinsic coherence of the kHz QPOs, and on the other the QPOs amplitude and quality factor in these two sources.