We demonstrate cooling of ultrathin fiber tapers coupled with nitrogen vacancy (NV) centers in nanodiamonds to cryogenic temperatures. Nanodiamonds containing multiple NV centers are deposited on the subwavelength 480-nm-diameter nanofiber region of fiber tapers. The fiber tapers are successfully cooled to 9 K using our home-built mounting holder and an optimized cooling speed. The fluorescence from the nanodiamond NV centers is efficiently channeled into a single guided mode and shows characteristic sharp zero-phonon lines of both neutral and negatively charged NV centers. The present nanofiber/nanodiamond hybrid systems at cryogenic temperatures can be used as NV-based quantum information devices and for highly sensitive nanoscale magnetometry in a cryogenic environment.