Laser-cooled atoms coupled to nanophotonic structures constitute a powerful research platform for the exploration of new regimes of light-matter interaction. While the initialization of the atomic internal degrees of freedom in these systems has been achieved, a full preparation of the atomic quantum state also requires controlling the center of mass motion of the atoms at the quantum level. Obtaining such control is not straightforward, due to the close vicinity of the atoms to the photonic system that is at ambient temperature. Here, we demonstrate cooling of individual neutral Cesium atoms, that are optically interfaced with light in an optical nanofiber, preparing them close to their three-dimensional motional ground state. The atoms are localized less than 300nm away from the hot fiber surface. Ground-state preparation is achieved by performing degenerate Raman cooling, and the atomic temperature is inferred from the analysis of heterodyne fluorescence spectroscopy signals. Our cooling method can be implemented either with externally applied or guided light fields. Moreover, it relies on polarization gradients which naturally occur for strongly confined guided optical fields. Thus, this method can be implemented in any trap based on nanophotonic structures. Our results provide an ideal starting point for the study of novel effects such as light-induced self-organization, the measurement of novel optical forces, and the investigation of heat transfer at the nanoscale using quantum probes.