We detect millihertz quasi-periodic oscillations (mHz QPOs) using the Rossi X-ray Time Explorer (RXTE) from the atoll neutron-star (NS) low-mass X-ray binaries 4U 1608--52 and Aql X--1. From the analysis of all RXTE observations of 4U 1608--52 and Aql X--1, we find mHz QPOs with a significance level $>3sigma$ in 49 and 47 observations, respectively. The QPO frequency is constrained between $sim$ 4.2 and 13.4 mHz. These types of mHz QPOs have been interpreted as being the result of marginally stable nuclear burning of He on the NS surface. We also report the discovery of a downward frequency drift in three observations of 4U 1608--52, making it the third source that shows this behaviour. We only find strong evidence of frequency drift in one occasion in Aql X--1, probably because the observations were too short to measure a significant drift. Finally, the mHz QPOs are mainly detected when both sources are in the soft or intermediate states; the cases that show frequency drift only occur when the sources are in intermediate states. Our results are consistent with the phenomenology observed for the NS systems 4U 1636--53 and EXO 0748--676, suggesting that all four sources can reach the conditions for marginally stable burning of He on the NS surface. These conditions depend on the source state in the same manner in all four systems.